Oct 30 23:40:53.457960 ip-10-0-131-123 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:40:53.878348 ip-10-0-131-123 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:40:53.878348 ip-10-0-131-123 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:40:53.878348 ip-10-0-131-123 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:40:53.878348 ip-10-0-131-123 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:40:53.881896 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.881896 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881895    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.881896 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881899    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881902    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881905    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881909    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881911    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881914    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881916    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881919    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881921    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881924    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881926    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881929    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881931    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881934    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881936    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881939    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881942    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881945    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881947    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.881986 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881950    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881958    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881961    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881963    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881966    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881968    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881971    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881973    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881976    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881978    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881980    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881983    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881985    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881988    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881991    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881994    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.881997    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882000    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882002    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.882415 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882005    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882010    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882013    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882017    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882020    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882023    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882026    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882028    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882031    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882033    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882036    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882038    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882041    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882043    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882046    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882049    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882052    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882056    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882058    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882061    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.882864 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882064    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882066    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882069    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882071    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882074    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882076    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882080    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882084    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882087    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882090    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882092    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882095    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882098    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882100    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882103    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882105    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882108    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882110    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882112    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.883323 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882115    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882117    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882120    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882122    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882125    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882128    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882130    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882133    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882137    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882140    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882143    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882145    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882148    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882150    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882153    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882155    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882158    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882160    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882163    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882166    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.883765 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882168    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882171    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882174    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882177    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882179    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882182    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.882184    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883006    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883014    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883016    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883019    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883021    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883024    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883027    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883029    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883032    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883035    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883037    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883040    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883044    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.884225 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883047    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883049    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883054    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883058    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883062    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883065    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883068    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883070    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883073    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883076    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883078    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883082    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883084    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883087    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883089    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883092    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883095    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883098    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.884704 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883100    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883103    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883106    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883109    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883112    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883114    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883117    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883120    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883122    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883125    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883127    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883130    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883133    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883136    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883139    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883143    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883148    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883151    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883154    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.885251 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883157    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883159    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883162    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883164    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883167    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883169    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883172    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883175    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883177    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883179    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883182    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883185    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883187    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883189    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883192    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883195    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883197    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883200    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883202    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883205    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.885719 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883208    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883210    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883213    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883215    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883218    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883221    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883223    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883226    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883228    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883231    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883234    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883237    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883239    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883242    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883245    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883247    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883249    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883252    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883255    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883257    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.886285 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883260    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883262    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883265    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883267    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883270    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883273    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883275    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883277    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883280    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883282    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883285    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883287    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883290    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883292    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883295    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883297    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883841    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883848    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883852    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883855    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.887005 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883858    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883861    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883864    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883867    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883869    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883873    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883876    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883879    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883881    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883884    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883886    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883888    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883891    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883894    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883897    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883899    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883902    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883904    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883906    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.887499 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883909    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883913    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883917    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883920    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883923    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883926    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883928    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883931    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883934    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883936    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883939    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883942    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883944    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883947    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883949    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883952    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883954    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883958    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883960    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883963    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.887982 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883965    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883968    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883970    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883972    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883975    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883977    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883980    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883982    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883985    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883987    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883990    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883992    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883994    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883996    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.883999    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884002    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884004    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884007    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884009    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.888439 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884012    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884015    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884020    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884022    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884025    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884027    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884031    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884035    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884038    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884041    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884044    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884048    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884050    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884053    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884056    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884058    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884061    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884063    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884066    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.888899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884069    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884071    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884074    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884077    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884079    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884082    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884087    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884089    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884092    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884095    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884097    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884099    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884103    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884105    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884109    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884112    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884114    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884117    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884119    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.889359 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884122    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884124    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884127    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884130    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.884133    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886038    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886066    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886088    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886096    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886105    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886112    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886119    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886133    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886138    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886143    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886149    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886155    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886160    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886164    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886169    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886174    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886183    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886190    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886197    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:40:53.889831 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886202    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886207    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886212    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886218    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886224    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886234    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886239    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886245    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886250    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886255    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886260    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886265    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886269    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886282    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886288    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886293    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886298    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886303    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886308    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886317    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886322    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886327    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886338    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886343    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:40:53.890414 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886350    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886354    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886360    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886365    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886370    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886376    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886386    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886392    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886397    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886402    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886407    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886414    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886419    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886425    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886431    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886441    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886446    2577 flags.go:64] FLAG: --help="false"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886450    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886456    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886461    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886466    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886474    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886479    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:40:53.891034 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886488    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886492    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886497    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886502    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886507    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886512    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886516    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886521    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886526    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886535    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886540    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886545    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886549    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886554    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886559    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886568    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886572    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886582    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886587    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886592    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886597    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886602    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886607    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886614    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886625    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:40:53.891564 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886645    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886651    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886656    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886661    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886666    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886671    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886675    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886680    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886697    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886702    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886732    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886857    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886863    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886868    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886871    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886874    2577 flags.go:64] FLAG: --port="10250"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886877    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886880    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-09a18c6d6c5eaa424"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886883    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886887    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886890    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886892    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886897    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886900    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886903    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886906    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:40:53.892164 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886909    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886912    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886915    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886918    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886922    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886925    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886928    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886931    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886933    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886936    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886939    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886942    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886945    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886948    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886951    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886954    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886957    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886961    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886964    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886967    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886976    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886979    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886982    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886988    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886991    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:40:53.892789 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886994    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.886997    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887000    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887002    2577 flags.go:64] FLAG: --v="2"
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887007    2577 flags.go:64] FLAG: --version="false"
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887011    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887015    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887019    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887160    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887164    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887169    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887172    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887175    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887178    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887180    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887183    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887186    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887189    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887191    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887194    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887198    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.893360 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887203    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887207    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887210    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887212    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887215    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887217    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887220    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887222    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887225    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887228    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887230    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887233    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887236    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887238    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887241    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887243    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887249    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887252    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887255    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887258    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.893863 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887260    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887264    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887267    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887269    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887272    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887275    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887277    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887280    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887282    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887285    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887287    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887290    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887292    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887295    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887297    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887300    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887302    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887305    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887307    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887310    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.894363 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887313    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887316    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887318    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887321    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887323    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887326    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887328    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887330    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887334    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887337    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887339    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887342    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887345    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887349    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887351    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887354    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887356    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887359    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887362    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887364    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.894830 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887367    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887369    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887372    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887374    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887377    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887379    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887382    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887384    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887387    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887390    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887393    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887395    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887398    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887400    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887403    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887405    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887407    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887410    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887412    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.895305 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887415    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887420    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887423    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887426    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887428    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887431    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887435    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887438    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887440    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887443    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887445    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887448    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887450    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887453    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.887480    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:40:53.895764 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887516    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887521    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887524    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887527    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887529    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887532    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887534    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887537    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887539    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887542    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887545    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887547    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887549    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887552    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887554    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887557    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887559    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887562    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887564    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887568    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.896110 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887571    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887574    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887576    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887580    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887583    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887585    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887588    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887590    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887593    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887595    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887598    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887600    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887603    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887605    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887608    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887610    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887613    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887615    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887617    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887620    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.896562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887623    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887625    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887628    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887648    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887651    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887654    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887656    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887659    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887661    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887664    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887666    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887671    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887673    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887676    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887678    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887681    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887684    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887687    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887690    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887692    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.897076 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887695    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887699    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887703    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887707    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887709    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887712    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887714    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887717    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887720    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887722    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887725    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887727    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887730    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887732    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887735    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887738    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887741    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887743    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887746    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.897526 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887748    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887751    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887753    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887755    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887759    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887762    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887764    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887768    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887772    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887775    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887777    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887780    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887783    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887786    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887788    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887792    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887794    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887798    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887800    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.897975 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887803    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887805    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887808    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887810    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887812    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887815    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887818    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.887820    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.895996    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.896012    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.896028    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.896033    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896091    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896096    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896101    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896105    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896109    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896112    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896115    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896118    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.898421 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896121    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896123    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896126    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896129    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896131    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896134    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896137    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896139    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896142    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896144    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896147    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896149    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896152    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896154    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896157    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896159    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896162    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896164    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896167    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896169    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.898891 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896172    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896176    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896179    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896182    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896184    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896187    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896190    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896192    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896195    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896198    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896200    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896203    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896206    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896208    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896211    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896214    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896216    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896219    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896222    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896224    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.899366 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896227    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896230    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896234    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896236    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896239    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896241    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896244    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896246    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896249    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896251    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896254    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896256    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896259    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896262    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896265    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896267    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896270    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896273    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896275    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.899859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896277    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896280    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896282    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896285    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896287    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896290    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896292    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896294    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896297    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896299    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896302    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896304    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896307    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896309    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896312    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896314    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896317    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896319    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896322    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896325    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.900286 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896327    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896330    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896332    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896335    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896338    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896340    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896342    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896345    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896348    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896351    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896354    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896356    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896358    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896361    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896364    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896366    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896368    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896371    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.900859 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896373    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.896401    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896577    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896581    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896584    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896587    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896590    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896592    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896596    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896600    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896603    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896606    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896610    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896612    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896615    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.901267 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896617    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896620    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896622    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896625    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896627    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896647    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896651    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896653    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896659    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896662    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896665    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896668    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896670    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896673    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896675    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896678    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896681    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896683    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896685    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896688    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.901621 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896690    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896693    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896695    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896698    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896701    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896703    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896706    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896709    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896712    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896714    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896716    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896719    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896722    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896724    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896727    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896729    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896731    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896734    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896737    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.902101 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896739    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896742    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896745    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896748    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896751    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896753    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896756    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896758    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896760    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896763    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896766    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896768    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896771    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896773    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896776    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896778    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896781    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896783    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896785    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.902562 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896788    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896790    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896793    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896795    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896798    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896800    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896803    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896806    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896810    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896813    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896817    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896819    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896822    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896825    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896827    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896829    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896833    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896835    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896838    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.903020 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896840    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896842    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896845    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896847    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896850    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896852    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896855    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896857    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896859    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896862    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896864    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896867    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896869    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896871    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896874    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896876    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.903450 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.896901    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896936    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896940    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896944    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896947    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896949    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896952    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896954    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896957    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896959    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896962    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896964    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896969    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896972    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896975    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896980    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896983    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896985    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896987    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:40:53.903869 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896990    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896992    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896995    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.896997    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897000    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897002    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897004    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897007    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897009    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897011    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897014    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897017    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897019    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897022    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897024    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897027    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897029    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897032    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897035    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:40:53.904452 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897037    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897040    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897042    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897044    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897047    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897050    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897053    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897055    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897058    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897060    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897063    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897065    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897068    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897070    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897072    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897075    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897079    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897084    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897088    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:40:53.904899 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897091    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897094    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897097    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897100    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897103    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897105    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897108    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897110    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897113    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897115    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897118    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897121    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897123    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897125    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897128    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897130    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897133    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897135    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897138    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897140    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:40:53.905343 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897142    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897145    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897148    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897150    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897153    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897155    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897158    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897160    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897165    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897170    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897173    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897178    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897181    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897185    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897187    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897190    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897192    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897195    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897198    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:40:53.905847 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897200    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897202    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897205    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897207    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897210    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897212    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897215    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897217    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897220    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:53.897222    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.897273    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.899676    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.900758    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.900857    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.901570    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.903019    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:40:53.906296 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.904113    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:40:53.916097 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.916078    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:40:53.919893 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.919877    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 23:40:53.920693 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.920671    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:40:53.923553 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.923534    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a09f4bc-237f-4cc4-b12d-89d266cd1d40:/dev/nvme0n1p3 9132e87a-039d-44cd-9453-9b15e1e01ab3:/dev/nvme0n1p4]
Oct 30 23:40:53.923644 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.923552    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 23:40:53.929923 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.929816    2577 manager.go:217] Machine: {Timestamp:2026-10-30 23:40:53.928288351 +0000 UTC m=+0.360024853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182702 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c979b3759a4f8432d33934e679e75 SystemUUID:ec2c979b-3759-a4f8-432d-33934e679e75 BootID:aa6428a3-06f4-4cda-ab80-5db31089f211 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c2:d8:cc:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c2:d8:cc:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:2f:e5:ac:db:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:40:53.929923 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.929921    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:40:53.930051 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.930039    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:40:53.933605 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.933579    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:40:53.933786 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.933606    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-123.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:40:53.933864 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.933791    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:40:53.933864 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.933804    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:40:53.933864 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.933861    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:40:53.933979 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.933883    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:40:53.935247 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.935234    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:40:53.939988 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.939973    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:40:53.940137 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.940126    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:40:53.940940 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.940929    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:40:53.941005 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.940946    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:40:53.941005 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.940960    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:40:53.941005 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.940973    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:40:53.941005 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.940985    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:40:53.941345 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.941332    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:40:53.943879 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.943857    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:40:53.951392 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.951371    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:40:53.952375 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952361    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:40:53.952446 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952400    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:40:53.952604 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952590    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:40:53.952675 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952612    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:40:53.952675 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952622    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:40:53.952675 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952646    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:40:53.952675 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952655    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:40:53.952675 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.952665    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:40:53.953367 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.953357    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:40:53.953417 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.953371    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:40:53.953417 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.953383    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:40:53.953417 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.953393    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:40:53.953417 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.953408    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:40:53.953579 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.953421    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:40:53.954212 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.954202    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:40:53.954261 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.954216    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:40:53.958267 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.958250    2577 server.go:1263] "Started kubelet"
Oct 30 23:40:53.958353 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.958301    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:40:53.958485 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.958442    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:40:53.958525 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.958514    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:40:53.959083 ip-10-0-131-123 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:40:53.962185 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:53.962171    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:40:53.964663 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:53.964622    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 23:40:54.066402 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.066376    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:40:54.072130 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.072109    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:40:54.072804 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.072784    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:40:54.073048 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.073024    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found"
Oct 30 23:40:54.074874 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.073752    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:40:54.074874 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.074000    2577 factory.go:55] Registering systemd factory
Oct 30 23:40:54.074874 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.074017    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:40:54.075072 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075052    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:40:54.075143 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075125    2577 factory.go:153] Registering CRI-O factory
Oct 30 23:40:54.075183 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075148    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:40:54.075224 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075202    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:40:54.075224 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075218    2577 factory.go:103] Registering Raw factory
Oct 30 23:40:54.075224 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075133    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:40:54.075327 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075230    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:40:54.075369 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075354    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:40:54.075369 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.075366    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:40:54.076177 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.076162    2577 manager.go:319] Starting recovery of all containers
Oct 30 23:40:54.086598 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.086581    2577 manager.go:324] Recovery completed
Oct 30 23:40:54.091890 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.091875    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.093736 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.093708    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.093813 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.093752    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.093813 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.093770    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.094365 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.094351    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:40:54.094365 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.094364    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:40:54.094485 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.094358    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.094485 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.094385    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:40:54.095140 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.095124    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.095199 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.095149    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.095199 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.095159    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.095419 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.095403    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-123.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 23:40:54.096512 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.095406    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-123.us-west-2.compute.internal.18e3731f15112416  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-123.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-123.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:40:53.958214678 +0000 UTC m=+0.389951179,LastTimestamp:2026-10-30 23:40:53.958214678 +0000 UTC m=+0.389951179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-123.us-west-2.compute.internal,}"
Oct 30 23:40:54.096734 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.096722    2577 policy_none.go:50] "Start"
Oct 30 23:40:54.096785 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.096738    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:40:54.096785 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.096751    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:40:54.098626 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.098616    2577 policy_none.go:44] "Start"
Oct 30 23:40:54.145867 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.141134    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:40:54.145867 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.142223    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:40:54.145867 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.142242    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:40:54.145867 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.142262    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:40:54.145867 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.142293    2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:40:54.146535 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.146509    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:40:54.148813 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.148796    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:40:54.148904 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.148825    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:40:54.148904 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.148833    2577 server.go:87] "Starting device plugin registration server"
Oct 30 23:40:54.149083 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.149070    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:40:54.149137 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.149083    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:40:54.149250 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.149216    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:40:54.149351 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.149295    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:40:54.149351 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.149302    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:40:54.149850 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.149829    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:40:54.149930 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.149886    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-123.us-west-2.compute.internal\" not found"
Oct 30 23:40:54.160374 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.160348    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:40:54.160476 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.160402    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccqnr"
Oct 30 23:40:54.243074 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.243033    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"]
Oct 30 23:40:54.243233 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.243107    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.244094 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.244079    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.244156 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.244109    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.244156 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.244120    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.246354 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.246339    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.246484 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.246470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.246518 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.246506    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.247093 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247070    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.247185 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247096    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.247185 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247105    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.247185 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247161    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.247388 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247186    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.247388 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247203    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.247468 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.247425    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:54.249357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.249340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.249446 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.249363    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.249446 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.249368    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:40:54.250075 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250057    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.250154 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250093    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.250154 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250118    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:40:54.250154 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250124    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.250278 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250141    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:40:54.250278 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250166    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.250278 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.250181    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:40:54.276210 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.276183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.276325 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.276217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.276325 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.276234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e32b577676385ef454846283b75daa6e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"e32b577676385ef454846283b75daa6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.281327 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.281302    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.283894 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.283879    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.338807 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.338780    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:54.338807 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.338812    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:54.340532 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.340511    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:54.340679 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.340625    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.340679 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.340672    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-123.us-west-2.compute.internal\": node \"ip-10-0-131-123.us-west-2.compute.internal\" not found"
Oct 30 23:40:54.373685 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.373663    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.376886 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.376870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.376951 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.376894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.376951 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.376912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e32b577676385ef454846283b75daa6e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"e32b577676385ef454846283b75daa6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.377047 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.376954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.377047 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.376969    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e32b577676385ef454846283b75daa6e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"e32b577676385ef454846283b75daa6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.377047 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.376998    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.435128 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.435060    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:40:54.435246 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.435168    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.490011 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.489982    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:40:54.587612 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.587428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.589758 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.589728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:40:54.900839 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.900800    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 23:40:54.901427 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.900989    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:40:54.901427 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.901013    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:40:54.901427 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.900989    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:40:54.901427 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:54.901027    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ade377f6ef8974a73ad19fd7b7959d61-5b50151745721f56.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-123.us-west-2.compute.internal.18e3731f1d250489\": read tcp 10.0.131.123:42760->3.143.20.58:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-123.us-west-2.compute.internal.18e3731f1d250489  default   5120 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-123.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-123.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-123.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:40:54 +0000 UTC,LastTimestamp:2026-10-30 23:40:54.247086524 +0000 UTC m=+0.678823027,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-123.us-west-2.compute.internal,}"
Oct 30 23:40:54.943835 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:54.943799    2577 apiserver.go:52] "Watching apiserver"
Oct 30 23:40:55.072735 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.072709    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 23:40:55.167070 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.166932    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:35:54 +0000 UTC" deadline="2028-07-02 10:08:56.197760835 +0000 UTC"
Oct 30 23:40:55.167070 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.166992    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14650h28m1.030771794s"
Oct 30 23:40:55.219626 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.219595    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:40:55.220010 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.219984    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-d7f6p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p","openshift-cluster-node-tuning-operator/tuned-9npkw","openshift-dns/node-resolver-kmhdt","openshift-image-registry/node-ca-ld74g","openshift-multus/network-metrics-daemon-hrmp7","openshift-network-diagnostics/network-check-target-t5xw7","openshift-network-operator/iptables-alerter-7xjgb","kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-j7mw9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal","openshift-multus/multus-h2w25","openshift-ovn-kubernetes/ovnkube-node-mdhf9","kube-system/global-pull-secret-syncer-gl7bk"]
Oct 30 23:40:55.222526 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.222507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.224725 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.224707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.227036 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.227020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.229123 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.229109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.231273 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.231256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.233544 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.233525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.233630 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.233610    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:40:55.235923 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.235901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:55.236040 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.235959    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:40:55.238284 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.238263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.240726 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.240696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.242988 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.242967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.245347 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.245328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.247657 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.247636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.247789 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.247700    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:40:55.275323 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.275298    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:40:55.276129 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.276106    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:40:55.276232 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.276113    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hcpbw\""
Oct 30 23:40:55.276869 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.276833    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:40:55.278024 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.278002    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.278024 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.278020    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.278186 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.278012    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7rnf\""
Oct 30 23:40:55.278237 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.278214    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:40:55.279818 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.279794    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.280203 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.280182    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72946\""
Oct 30 23:40:55.280398 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.280377    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.281104 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-bin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.281198 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bg7r\" (UniqueName: \"kubernetes.io/projected/24273fe9-3b2a-421c-bec7-26c6af951dcb-kube-api-access-7bg7r\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.281198 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-systemd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.281320 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-ovn\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.281320 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-env-overrides\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.281320 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-device-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.281460 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-daemon-config\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.281460 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-log-socket\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.281536 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-sys\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.281536 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-tuned\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.281536 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.281633 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-conf-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.281633 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysconfig\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.281633 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-systemd\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqsst\" (UniqueName: \"kubernetes.io/projected/685220ca-ec2c-4955-91eb-53a4d5a9c62b-kube-api-access-pqsst\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grks\" (UniqueName: \"kubernetes.io/projected/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kube-api-access-2grks\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281732    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-netns\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281757    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281781    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ksfp2\""
Oct 30 23:40:55.281800 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-kubelet\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-lib-modules\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdcp\" (UniqueName: \"kubernetes.io/projected/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-kube-api-access-jjdcp\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-socket-dir-parent\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.281969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-config\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.282076 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-modprobe-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-var-lib-kubelet\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-system-cni-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d645c1da-d44f-4765-bcdf-afe19821b63c-serviceca\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqds5\" (UniqueName: \"kubernetes.io/projected/d645c1da-d44f-4765-bcdf-afe19821b63c-kube-api-access-qqds5\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9h7\" (UniqueName: \"kubernetes.io/projected/a7f8297f-f076-43cb-ba9f-93a455101c9f-kube-api-access-wd9h7\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:55.282326 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-var-lib-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-agent-certs\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-k8s-cni-cncf-io\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-slash\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-script-lib\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-dbus\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-host\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97c08a18-b069-47a5-bdfd-7fca0c7767ae-host-slash\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/97c08a18-b069-47a5-bdfd-7fca0c7767ae-kube-api-access-tlm89\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.282646 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-multus\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-multus-certs\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-etc-kubernetes\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-etc-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qrj\" (UniqueName: \"kubernetes.io/projected/f6b0becd-1e63-4dc2-a01e-3415794c4b46-kube-api-access-68qrj\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovn-node-metrics-cert\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-kubernetes\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-conf\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-bin\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97c08a18-b069-47a5-bdfd-7fca0c7767ae-iptables-alerter-script\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.282989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d645c1da-d44f-4765-bcdf-afe19821b63c-host\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvvz\" (UniqueName: \"kubernetes.io/projected/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-kube-api-access-fjvvz\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-kubelet\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283126 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-systemd-units\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-netns\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-node-log\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-netd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-tmp\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685220ca-ec2c-4955-91eb-53a4d5a9c62b-tmp-dir\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-cnibin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cnibin\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-hostroot\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-kubelet-config\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-run\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.283744 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-konnectivity-ca\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/685220ca-ec2c-4955-91eb-53a4d5a9c62b-hosts-file\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-system-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-cni-binary-copy\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283886    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwz57\""
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283943    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283959    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.284295 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.283985    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:40:55.291055 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.291017    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.291158 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.291105    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.291287 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.291274    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnlbs\""
Oct 30 23:40:55.291344 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.291310    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:40:55.380878 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.380849    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:40:55.381016 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.380911    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2lgtj"
Oct 30 23:40:55.384100 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-slash\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384162 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-script-lib\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384162 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-dbus\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.384162 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-host\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97c08a18-b069-47a5-bdfd-7fca0c7767ae-host-slash\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-slash\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/97c08a18-b069-47a5-bdfd-7fca0c7767ae-kube-api-access-tlm89\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-host\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384251    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97c08a18-b069-47a5-bdfd-7fca0c7767ae-host-slash\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384227    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-dbus\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.384299 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-multus\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-multus-certs\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-etc-kubernetes\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384368    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-multus\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-etc-kubernetes\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-multus-certs\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-etc-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qrj\" (UniqueName: \"kubernetes.io/projected/f6b0becd-1e63-4dc2-a01e-3415794c4b46-kube-api-access-68qrj\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384467    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-etc-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovn-node-metrics-cert\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-kubernetes\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384495    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.384540 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-conf\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-bin\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-kubernetes\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384626    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-bin\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-conf\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97c08a18-b069-47a5-bdfd-7fca0c7767ae-iptables-alerter-script\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d645c1da-d44f-4765-bcdf-afe19821b63c-host\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384779    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d645c1da-d44f-4765-bcdf-afe19821b63c-host\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvvz\" (UniqueName: \"kubernetes.io/projected/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-kube-api-access-fjvvz\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-kubelet\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-kubelet\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-systemd-units\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.384989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-netns\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-node-log\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385012    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-systemd-units\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-netd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.385240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385048    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-tmp\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685220ca-ec2c-4955-91eb-53a4d5a9c62b-tmp-dir\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385105    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-node-log\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-cnibin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cnibin\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-netns\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385160    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385116    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97c08a18-b069-47a5-bdfd-7fca0c7767ae-iptables-alerter-script\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385258    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cnibin\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-hostroot\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-hostroot\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385304    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-cnibin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385328    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385341    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.386085 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385348    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-netd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385374    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-kubelet-config\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385401    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385405    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385259    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-run\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385461    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385474    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-kubelet-config\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385484    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385501    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385508    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685220ca-ec2c-4955-91eb-53a4d5a9c62b-tmp-dir\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-konnectivity-ca\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/685220ca-ec2c-4955-91eb-53a4d5a9c62b-hosts-file\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385601    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-run\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-system-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387060 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-cni-binary-copy\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385669    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/685220ca-ec2c-4955-91eb-53a4d5a9c62b-hosts-file\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385660    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-system-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385765    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385738    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-bin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385793    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hnph4\""
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385845    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xnb7d\""
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-bin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bg7r\" (UniqueName: \"kubernetes.io/projected/24273fe9-3b2a-421c-bec7-26c6af951dcb-kube-api-access-7bg7r\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385961    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zg485\""
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-systemd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.385993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-ovn\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-env-overrides\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-device-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386051    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-systemd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386096    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-ovn\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-daemon-config\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-log-socket\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.387993 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-sys\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-konnectivity-ca\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-tuned\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-cni-binary-copy\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-log-socket\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-conf-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-device-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386303    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysconfig\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386397    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-sys\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysconfig\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-conf-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-systemd\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsst\" (UniqueName: \"kubernetes.io/projected/685220ca-ec2c-4955-91eb-53a4d5a9c62b-kube-api-access-pqsst\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grks\" (UniqueName: \"kubernetes.io/projected/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kube-api-access-2grks\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-netns\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-env-overrides\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.388763 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-kubelet\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-lib-modules\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdcp\" (UniqueName: \"kubernetes.io/projected/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-kube-api-access-jjdcp\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-daemon-config\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-systemd\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386918    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-lib-modules\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-netns\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-kubelet\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.386999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-socket-dir-parent\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-config\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387077    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-modprobe-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-var-lib-kubelet\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.387176    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-system-cni-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.389404 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d645c1da-d44f-4765-bcdf-afe19821b63c-serviceca\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqds5\" (UniqueName: \"kubernetes.io/projected/d645c1da-d44f-4765-bcdf-afe19821b63c-kube-api-access-qqds5\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9h7\" (UniqueName: \"kubernetes.io/projected/a7f8297f-f076-43cb-ba9f-93a455101c9f-kube-api-access-wd9h7\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387454    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-system-cni-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-var-lib-kubelet\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-socket-dir-parent\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.387515    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-modprobe-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-var-lib-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.387602    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret podName:45dfef13-d5e6-4bff-98d2-47032ef6f624 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:55.887565925 +0000 UTC m=+2.319302437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret") pod "global-pull-secret-syncer-gl7bk" (UID: "45dfef13-d5e6-4bff-98d2-47032ef6f624") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-var-lib-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.387668    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:55.887637687 +0000 UTC m=+2.319374184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-agent-certs\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.390357 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-k8s-cni-cncf-io\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387836    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387945    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-k8s-cni-cncf-io\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.387932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d645c1da-d44f-4765-bcdf-afe19821b63c-serviceca\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.388030    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-config\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.388553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.391262 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.389383    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-tmp\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.395706 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.395683    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-tuned\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.396056 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.396039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-agent-certs\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.396848 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.396820    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.396916 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.396895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-script-lib\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.397012 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.396997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.397442 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.397422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovn-node-metrics-cert\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.447170 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.447098    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:40:55.447170 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.447127    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:40:55.447170 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.447145    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zgxnf for pod openshift-network-diagnostics/network-check-target-t5xw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:55.447170 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.447148    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/97c08a18-b069-47a5-bdfd-7fca0c7767ae-kube-api-access-tlm89\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.447488 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.447203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvvz\" (UniqueName: \"kubernetes.io/projected/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-kube-api-access-fjvvz\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.447488 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.447257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qrj\" (UniqueName: \"kubernetes.io/projected/f6b0becd-1e63-4dc2-a01e-3415794c4b46-kube-api-access-68qrj\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.447488 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.447273    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf podName:3b67fb8d-7158-4441-9f05-5bd7aebdcf5c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:55.947252973 +0000 UTC m=+2.378989478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zgxnf" (UniqueName: "kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf") pod "network-check-target-t5xw7" (UID: "3b67fb8d-7158-4441-9f05-5bd7aebdcf5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:55.447488 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.447384    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bg7r\" (UniqueName: \"kubernetes.io/projected/24273fe9-3b2a-421c-bec7-26c6af951dcb-kube-api-access-7bg7r\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.448376 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.448356    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdcp\" (UniqueName: \"kubernetes.io/projected/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-kube-api-access-jjdcp\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.452067 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.452042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsst\" (UniqueName: \"kubernetes.io/projected/685220ca-ec2c-4955-91eb-53a4d5a9c62b-kube-api-access-pqsst\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.452381 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.452357    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grks\" (UniqueName: \"kubernetes.io/projected/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kube-api-access-2grks\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.452617 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.452600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9h7\" (UniqueName: \"kubernetes.io/projected/a7f8297f-f076-43cb-ba9f-93a455101c9f-kube-api-access-wd9h7\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.453320 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.453298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqds5\" (UniqueName: \"kubernetes.io/projected/d645c1da-d44f-4765-bcdf-afe19821b63c-kube-api-access-qqds5\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.555653 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.555618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:40:55.580429 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.580400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:40:55.581567 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.581543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:40:55.588703 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.588684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:40:55.597951 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.597934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:40:55.613750 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.613732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:40:55.632506 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.632480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:55.642638 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.642619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h2w25"
Oct 30 23:40:55.673952 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.673913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:40:55.892089 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.892058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:55.892375 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.892093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:55.892375 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.892206    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:55.892375 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.892206    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:55.892375 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.892262    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret podName:45dfef13-d5e6-4bff-98d2-47032ef6f624 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:56.892246924 +0000 UTC m=+3.323983433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret") pod "global-pull-secret-syncer-gl7bk" (UID: "45dfef13-d5e6-4bff-98d2-47032ef6f624") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:55.892375 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.892278    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:56.89226944 +0000 UTC m=+3.324005931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:55.992985 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:55.992952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:55.993526 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.993118    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:40:55.993526 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.993148    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:40:55.993526 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.993163    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zgxnf for pod openshift-network-diagnostics/network-check-target-t5xw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:55.993526 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:55.993234    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf podName:3b67fb8d-7158-4441-9f05-5bd7aebdcf5c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:56.993216032 +0000 UTC m=+3.424952524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zgxnf" (UniqueName: "kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf") pod "network-check-target-t5xw7" (UID: "3b67fb8d-7158-4441-9f05-5bd7aebdcf5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:56.013957 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.013922    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f30de5e28d2739412bf74e36d5e3bf9.slice/crio-9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553 WatchSource:0}: Error finding container 9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553: Status 404 returned error can't find the container with id 9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553
Oct 30 23:40:56.014341 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.014323    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32b577676385ef454846283b75daa6e.slice/crio-85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f WatchSource:0}: Error finding container 85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f: Status 404 returned error can't find the container with id 85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f
Oct 30 23:40:56.019163 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.019147    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:40:56.147551 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.147443    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"6f30de5e28d2739412bf74e36d5e3bf9","Type":"ContainerStarted","Data":"9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553"}
Oct 30 23:40:56.148488 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.148465    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"e32b577676385ef454846283b75daa6e","Type":"ContainerStarted","Data":"85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f"}
Oct 30 23:40:56.158553 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.158527    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:56.278978 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.278947    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:56.383794 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.383726    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:35:55 +0000 UTC" deadline="2028-04-02 04:18:45.60808304 +0000 UTC"
Oct 30 23:40:56.383794 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.383785    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12460h37m49.224300925s"
Oct 30 23:40:56.496029 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.495999    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685220ca_ec2c_4955_91eb_53a4d5a9c62b.slice/crio-a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527 WatchSource:0}: Error finding container a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527: Status 404 returned error can't find the container with id a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527
Oct 30 23:40:56.499581 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.499564    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:40:56.519544 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.519515    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555bcb1e_9ce9_46e1_ab6b_c4cb08127e9b.slice/crio-6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a WatchSource:0}: Error finding container 6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a: Status 404 returned error can't find the container with id 6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a
Oct 30 23:40:56.535652 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.535623    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c08a18_b069_47a5_bdfd_7fca0c7767ae.slice/crio-159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9 WatchSource:0}: Error finding container 159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9: Status 404 returned error can't find the container with id 159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9
Oct 30 23:40:56.563188 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.563160    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode895adf2_43c1_46f2_b2de_a3bddfd08f0c.slice/crio-31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696 WatchSource:0}: Error finding container 31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696: Status 404 returned error can't find the container with id 31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696
Oct 30 23:40:56.600204 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.600172    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd645c1da_d44f_4765_bcdf_afe19821b63c.slice/crio-1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb WatchSource:0}: Error finding container 1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb: Status 404 returned error can't find the container with id 1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb
Oct 30 23:40:56.630942 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.630911    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24273fe9_3b2a_421c_bec7_26c6af951dcb.slice/crio-4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82 WatchSource:0}: Error finding container 4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82: Status 404 returned error can't find the container with id 4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82
Oct 30 23:40:56.655028 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.654999    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f8297f_f076_43cb_ba9f_93a455101c9f.slice/crio-e10e9c9355e6c2a3c025d82f772e376086706e7093d491ab658c0be24064d27a WatchSource:0}: Error finding container e10e9c9355e6c2a3c025d82f772e376086706e7093d491ab658c0be24064d27a: Status 404 returned error can't find the container with id e10e9c9355e6c2a3c025d82f772e376086706e7093d491ab658c0be24064d27a
Oct 30 23:40:56.685554 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:40:56.685526    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b0becd_1e63_4dc2_a01e_3415794c4b46.slice/crio-df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208 WatchSource:0}: Error finding container df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208: Status 404 returned error can't find the container with id df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208
Oct 30 23:40:56.898908 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.898875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:56.899089 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.898916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:56.899089 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.899038    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:56.899164 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.899095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret podName:45dfef13-d5e6-4bff-98d2-47032ef6f624 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:58.899080187 +0000 UTC m=+5.330816678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret") pod "global-pull-secret-syncer-gl7bk" (UID: "45dfef13-d5e6-4bff-98d2-47032ef6f624") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:56.899164 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.899038    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:56.899164 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.899130    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:40:58.89911944 +0000 UTC m=+5.330855931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:56.999273 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:56.999242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:56.999772 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.999380    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:40:56.999772 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.999394    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:40:56.999772 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.999405    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zgxnf for pod openshift-network-diagnostics/network-check-target-t5xw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:56.999772 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:56.999455    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf podName:3b67fb8d-7158-4441-9f05-5bd7aebdcf5c nodeName:}" failed. No retries permitted until 2026-10-30 23:40:58.999438907 +0000 UTC m=+5.431175414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zgxnf" (UniqueName: "kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf") pod "network-check-target-t5xw7" (UID: "3b67fb8d-7158-4441-9f05-5bd7aebdcf5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:57.143288 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.143250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:57.143288 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.143276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:57.143288 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.143276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:57.143533 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:57.143384    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:40:57.143533 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:57.143446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:40:57.143533 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:57.143503    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:40:57.151930 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.151854    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208"}
Oct 30 23:40:57.152946 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.152919    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h2w25" event={"ID":"24273fe9-3b2a-421c-bec7-26c6af951dcb","Type":"ContainerStarted","Data":"4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82"}
Oct 30 23:40:57.154206 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.154161    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ld74g" event={"ID":"d645c1da-d44f-4765-bcdf-afe19821b63c","Type":"ContainerStarted","Data":"1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb"}
Oct 30 23:40:57.155490 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.155468    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7f6p" event={"ID":"e895adf2-43c1-46f2-b2de-a3bddfd08f0c","Type":"ContainerStarted","Data":"31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696"}
Oct 30 23:40:57.156730 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.156700    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xjgb" event={"ID":"97c08a18-b069-47a5-bdfd-7fca0c7767ae","Type":"ContainerStarted","Data":"159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9"}
Oct 30 23:40:57.157943 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.157922    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p" event={"ID":"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b","Type":"ContainerStarted","Data":"6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a"}
Oct 30 23:40:57.159104 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.159084    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9npkw" event={"ID":"a7f8297f-f076-43cb-ba9f-93a455101c9f","Type":"ContainerStarted","Data":"e10e9c9355e6c2a3c025d82f772e376086706e7093d491ab658c0be24064d27a"}
Oct 30 23:40:57.160102 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.160076    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kmhdt" event={"ID":"685220ca-ec2c-4955-91eb-53a4d5a9c62b","Type":"ContainerStarted","Data":"a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527"}
Oct 30 23:40:57.384628 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.384554    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:35:55 +0000 UTC" deadline="2028-08-04 06:43:32.076382246 +0000 UTC"
Oct 30 23:40:57.384628 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:57.384625    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15439h2m34.691762052s"
Oct 30 23:40:58.095567 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.095412    2577 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_multus-additional-cni-plugins-j7mw9_openshift-multus_44fe22ce-a102-4f4f-9f42-6c80dae9c9d6_0\": unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/16/16ed6d68be8f0dfe28633c57bbc0e6240d3e1702958dc74bc992070d975c2bac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T234057Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=0a76df68198a6ce9343c1fc1d7f24d5a32a13183407f01a2b951b5f9e396c07d&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793404557~hmac=7ca0577840648bd2c9ef2e5f44c4b05efe94ec060bea8b8d5bc986aba23b4138\": EOF"
Oct 30 23:40:58.095567 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.095503    2577 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_multus-additional-cni-plugins-j7mw9_openshift-multus_44fe22ce-a102-4f4f-9f42-6c80dae9c9d6_0\": unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/16/16ed6d68be8f0dfe28633c57bbc0e6240d3e1702958dc74bc992070d975c2bac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T234057Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=0a76df68198a6ce9343c1fc1d7f24d5a32a13183407f01a2b951b5f9e396c07d&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793404557~hmac=7ca0577840648bd2c9ef2e5f44c4b05efe94ec060bea8b8d5bc986aba23b4138\": EOF" pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:58.095567 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.095540    2577 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_multus-additional-cni-plugins-j7mw9_openshift-multus_44fe22ce-a102-4f4f-9f42-6c80dae9c9d6_0\": unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/16/16ed6d68be8f0dfe28633c57bbc0e6240d3e1702958dc74bc992070d975c2bac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T234057Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=0a76df68198a6ce9343c1fc1d7f24d5a32a13183407f01a2b951b5f9e396c07d&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793404557~hmac=7ca0577840648bd2c9ef2e5f44c4b05efe94ec060bea8b8d5bc986aba23b4138\": EOF" pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:40:58.096193 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.095634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"multus-additional-cni-plugins-j7mw9_openshift-multus(44fe22ce-a102-4f4f-9f42-6c80dae9c9d6)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"multus-additional-cni-plugins-j7mw9_openshift-multus(44fe22ce-a102-4f4f-9f42-6c80dae9c9d6)\\\": rpc error: code = Unknown desc = creating pod sandbox with name \\\"k8s_multus-additional-cni-plugins-j7mw9_openshift-multus_44fe22ce-a102-4f4f-9f42-6c80dae9c9d6_0\\\": unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/16/16ed6d68be8f0dfe28633c57bbc0e6240d3e1702958dc74bc992070d975c2bac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T234057Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=0a76df68198a6ce9343c1fc1d7f24d5a32a13183407f01a2b951b5f9e396c07d&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793404557~hmac=7ca0577840648bd2c9ef2e5f44c4b05efe94ec060bea8b8d5bc986aba23b4138\\\": EOF\"" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" podUID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6"
Oct 30 23:40:58.916432 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:58.916389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:58.916648 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:58.916447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:58.916648 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.916623    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:58.916898 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.916712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret podName:45dfef13-d5e6-4bff-98d2-47032ef6f624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:02.916691068 +0000 UTC m=+9.348427573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret") pod "global-pull-secret-syncer-gl7bk" (UID: "45dfef13-d5e6-4bff-98d2-47032ef6f624") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:40:58.916898 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.916798    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:58.916898 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:58.916842    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:02.916828813 +0000 UTC m=+9.348565304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:40:59.018013 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:59.017684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:59.018013 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.017865    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:40:59.018013 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.017891    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:40:59.018013 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.017907    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zgxnf for pod openshift-network-diagnostics/network-check-target-t5xw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:59.018013 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.017972    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf podName:3b67fb8d-7158-4441-9f05-5bd7aebdcf5c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:03.017951598 +0000 UTC m=+9.449688095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zgxnf" (UniqueName: "kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf") pod "network-check-target-t5xw7" (UID: "3b67fb8d-7158-4441-9f05-5bd7aebdcf5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:40:59.143672 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:59.143350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:40:59.143672 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.143489    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:40:59.143672 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:59.143566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:40:59.143672 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.143661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:40:59.144457 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:40:59.143725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:40:59.144457 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:40:59.143796    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:01.143316 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:01.143037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:01.143697 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:01.143450    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:01.143697 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:01.143546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:01.143697 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:01.143648    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:01.143885 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:01.143729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:01.143885 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:01.143807    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:01.198934 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:01.198890    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f30de5e28d2739412bf74e36d5e3bf9" containerID="c289690828346d4b9a49cf2e3e9ef145b61a8f85ee96af67c423e8fdca402f1f" exitCode=0
Oct 30 23:41:01.199106 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:01.199008    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"6f30de5e28d2739412bf74e36d5e3bf9","Type":"ContainerDied","Data":"c289690828346d4b9a49cf2e3e9ef145b61a8f85ee96af67c423e8fdca402f1f"}
Oct 30 23:41:01.216225 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:01.216187    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"e32b577676385ef454846283b75daa6e","Type":"ContainerStarted","Data":"a2002ecafe3437b62ab8b9885222cbe81ceecaa7b49823e855b69d31848fb01f"}
Oct 30 23:41:02.226235 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:02.226190    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"6f30de5e28d2739412bf74e36d5e3bf9","Type":"ContainerStarted","Data":"37f2a9884952984032c416d7571957c409ea6850a0ab24639b8a7c0c9eef393f"}
Oct 30 23:41:02.232288 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:02.232253    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7f6p" event={"ID":"e895adf2-43c1-46f2-b2de-a3bddfd08f0c","Type":"ContainerStarted","Data":"c7c5d83f4076c7bb53bb71180801fedf8eb9b196944473164c3e04aa19e6734a"}
Oct 30 23:41:02.343871 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:02.343804    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal" podStartSLOduration=7.343784167 podStartE2EDuration="7.343784167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:01.433550725 +0000 UTC m=+7.865287240" watchObservedRunningTime="2026-10-30 23:41:02.343784167 +0000 UTC m=+8.775520684"
Oct 30 23:41:02.344075 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:02.344016    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" podStartSLOduration=7.344006674 podStartE2EDuration="7.344006674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:02.343435101 +0000 UTC m=+8.775171616" watchObservedRunningTime="2026-10-30 23:41:02.344006674 +0000 UTC m=+8.775743191"
Oct 30 23:41:02.953776 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:02.953676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:02.953776 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:02.953735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:02.954061 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:02.953851    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:02.954061 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:02.953866    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:02.954061 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:02.953922    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret podName:45dfef13-d5e6-4bff-98d2-47032ef6f624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:10.953908491 +0000 UTC m=+17.385644996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret") pod "global-pull-secret-syncer-gl7bk" (UID: "45dfef13-d5e6-4bff-98d2-47032ef6f624") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:02.954061 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:02.953937    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:10.953930441 +0000 UTC m=+17.385666931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:03.059151 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:03.059113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:03.059325 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.059310    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:03.059383 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.059335    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:03.059383 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.059353    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zgxnf for pod openshift-network-diagnostics/network-check-target-t5xw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:03.059446 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.059424    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf podName:3b67fb8d-7158-4441-9f05-5bd7aebdcf5c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:11.059398584 +0000 UTC m=+17.491135076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zgxnf" (UniqueName: "kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf") pod "network-check-target-t5xw7" (UID: "3b67fb8d-7158-4441-9f05-5bd7aebdcf5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:03.143545 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:03.143499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:03.143729 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.143640    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:03.143784 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:03.143766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:03.143891 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.143870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:03.143949 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:03.143936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:03.144037 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:03.144018    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:04.240445 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:04.240404    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ld74g" event={"ID":"d645c1da-d44f-4765-bcdf-afe19821b63c","Type":"ContainerStarted","Data":"5f1f9ca0b1c188681e347e35fd062931e51ba325434c4b079db98990ed93577a"}
Oct 30 23:41:04.242950 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:04.242925    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p" event={"ID":"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b","Type":"ContainerStarted","Data":"96dcfd142a3fb78ee32efc7243d0b212f3f8265c577d550acdd000b7967b6681"}
Oct 30 23:41:04.357439 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:04.357369    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d7f6p" podStartSLOduration=4.865562455 podStartE2EDuration="9.357343439s" totalImagesPullingTime="4.491780984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:02.461734787 +0000 UTC m=+8.893471304" watchObservedRunningTime="2026-10-30 23:41:04.357343439 +0000 UTC m=+10.789079954"
Oct 30 23:41:04.357872 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:04.357826    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ld74g" podStartSLOduration=2.867109726 podStartE2EDuration="9.357810698s" totalImagesPullingTime="6.490700972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:04.357491157 +0000 UTC m=+10.789227690" watchObservedRunningTime="2026-10-30 23:41:04.357810698 +0000 UTC m=+10.789547214"
Oct 30 23:41:05.143665 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.143582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:05.143665 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.143611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:05.143665 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.143582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:05.143910 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:05.143723    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:05.143910 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:05.143827    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:05.144002 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:05.143903    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:05.255135 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.255101    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h2w25" event={"ID":"24273fe9-3b2a-421c-bec7-26c6af951dcb","Type":"ContainerStarted","Data":"c40e008c755a217ef3db063010202bd1006cd45430e874041f4bcb8780cfc010"}
Oct 30 23:41:05.258283 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.258248    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kmhdt" event={"ID":"685220ca-ec2c-4955-91eb-53a4d5a9c62b","Type":"ContainerStarted","Data":"5962563e23b04ef897f9c4844a472782aeca06846694330104bf278235f21817"}
Oct 30 23:41:05.316948 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.316929    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:41:05.475442 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.475346    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h2w25" podStartSLOduration=2.25987687 podStartE2EDuration="10.475330569s" totalImagesPullingTime="8.215453699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:05.367543373 +0000 UTC m=+11.799279885" watchObservedRunningTime="2026-10-30 23:41:05.475330569 +0000 UTC m=+11.907067081"
Oct 30 23:41:05.475854 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:05.475811    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kmhdt" podStartSLOduration=2.111929984 podStartE2EDuration="10.475800647s" totalImagesPullingTime="8.363870663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:05.474893435 +0000 UTC m=+11.906629948" watchObservedRunningTime="2026-10-30 23:41:05.475800647 +0000 UTC m=+11.907537157"
Oct 30 23:41:06.156549 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:06.155937    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:41:05.316945484Z","UUID":"87b444cb-1220-46f1-abf4-3abb81aec8de","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:41:06.158239 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:06.158082    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:41:06.158239 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:06.158112    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:41:06.263322 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:06.263247    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p" event={"ID":"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b","Type":"ContainerStarted","Data":"0cc6dcdc19391ed797de9589e532c94c4322e2417240112ee077854ac8faeb4d"}
Oct 30 23:41:06.706316 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:06.706286    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:06.707730 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:06.707693    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:07.142776 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.142747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:07.142776 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.142768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:07.142986 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.142874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:07.142986 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:07.142876    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:07.142986 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:07.142967    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:07.143147 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:07.143039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:07.266650 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.266458    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9npkw" event={"ID":"a7f8297f-f076-43cb-ba9f-93a455101c9f","Type":"ContainerStarted","Data":"16490bf42aff086574912b9c3e2304d7f114ecfd46ccf15e6c0b1ec17c3cd41f"}
Oct 30 23:41:07.268317 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.268289    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xjgb" event={"ID":"97c08a18-b069-47a5-bdfd-7fca0c7767ae","Type":"ContainerStarted","Data":"d1059e3ec57d1e29cfec1934b72f0d15d035347560030a2bc8b7ba18c3aa6a01"}
Oct 30 23:41:07.270562 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.270541    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p" event={"ID":"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b","Type":"ContainerStarted","Data":"99c9d57090b38def0b77e42e4f494d38595bd7db8406d6efac5d2565768d5d92"}
Oct 30 23:41:07.377148 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.377092    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9npkw" podStartSLOduration=2.213202145 podStartE2EDuration="12.377067862s" totalImagesPullingTime="10.163865717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:07.376768864 +0000 UTC m=+13.808505375" watchObservedRunningTime="2026-10-30 23:41:07.377067862 +0000 UTC m=+13.808804373"
Oct 30 23:41:07.485961 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.485869    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7xjgb" podStartSLOduration=3.873454404 podStartE2EDuration="12.485852663s" totalImagesPullingTime="8.612398259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:07.485377081 +0000 UTC m=+13.917113590" watchObservedRunningTime="2026-10-30 23:41:07.485852663 +0000 UTC m=+13.917589177"
Oct 30 23:41:07.596028 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:07.595982    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p" podStartSLOduration=2.641322039 podStartE2EDuration="12.595971335s" totalImagesPullingTime="9.954649296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:07.59561971 +0000 UTC m=+14.027356222" watchObservedRunningTime="2026-10-30 23:41:07.595971335 +0000 UTC m=+14.027707845"
Oct 30 23:41:09.143345 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:09.143314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:09.143345 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:09.143348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:09.144098 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:09.143318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:09.144098 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:09.143435    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:09.144098 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:09.143317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:09.144098 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:09.143547    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:09.144098 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:09.143803    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:09.144098 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:09.143826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:09.155180 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:41:09.155153    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fe22ce_a102_4f4f_9f42_6c80dae9c9d6.slice/crio-ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3 WatchSource:0}: Error finding container ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3: Status 404 returned error can't find the container with id ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3
Oct 30 23:41:09.275627 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:09.275593    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerStarted","Data":"ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3"}
Oct 30 23:41:11.021947 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.021914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:11.022626 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.021952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:11.022626 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.022061    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:11.022626 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.022062    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:11.022626 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.022117    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret podName:45dfef13-d5e6-4bff-98d2-47032ef6f624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.02210131 +0000 UTC m=+33.453837812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret") pod "global-pull-secret-syncer-gl7bk" (UID: "45dfef13-d5e6-4bff-98d2-47032ef6f624") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:11.022626 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.022184    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.022162053 +0000 UTC m=+33.453898542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:11.122401 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.122368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:11.122571 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.122484    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:11.122571 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.122497    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:11.122571 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.122506    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zgxnf for pod openshift-network-diagnostics/network-check-target-t5xw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:11.122571 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.122558    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf podName:3b67fb8d-7158-4441-9f05-5bd7aebdcf5c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.122541431 +0000 UTC m=+33.554277941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zgxnf" (UniqueName: "kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf") pod "network-check-target-t5xw7" (UID: "3b67fb8d-7158-4441-9f05-5bd7aebdcf5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:11.143337 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.143311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:11.143484 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.143387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:11.143544 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.143502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:11.143544 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.143507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:11.143666 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.143622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:11.143770 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:11.143746    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:11.280757 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.280723    2577 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="a60124edde9bc20711720a321fe62ffa0191bbf18b9a3abbddfe2d655d2c3458" exitCode=0
Oct 30 23:41:11.280919 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:11.280780    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerDied","Data":"a60124edde9bc20711720a321fe62ffa0191bbf18b9a3abbddfe2d655d2c3458"}
Oct 30 23:41:12.286256 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:12.286036    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"55d1e2fc61e9090ff61a35fe57c3479e9589721fc3a7ece513f68ae66cfbae40"}
Oct 30 23:41:12.286256 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:12.286254    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"e4b69494d110c41df491253da68f2e82c8e348e30aecff2626d4c4ccf6441355"}
Oct 30 23:41:12.286647 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:12.286285    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"f7409ec66dfc7abbd7f0b7551d2ad41bc8e9a6926f6a407ac3841e3c8d64c80b"}
Oct 30 23:41:12.286647 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:12.286311    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"80f55caca3182db6414954408df7380235bd8b81ba67e7d6200d2f402fc370cf"}
Oct 30 23:41:12.286647 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:12.286337    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"72856bdb3180ad66080c5282ed7a1d92bbe5b103453d616252144a7555f17834"}
Oct 30 23:41:12.286647 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:12.286362    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"80e698fada62e8972009a68631ea2e57ae4e8fc8bfeb213428fb3fbc02cc7ec8"}
Oct 30 23:41:13.142649 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:13.142608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:13.142820 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:13.142608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:13.142820 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:13.142730    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:13.142820 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:13.142608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:13.142944 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:13.142825    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:13.142944 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:13.142888    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:14.292718 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:14.292477    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"499ad22ff87449d1e69842316eb9517766a1879a60f836e7d02a78075db35668"}
Oct 30 23:41:14.294294 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:14.294268    2577 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="4bd45d2ac8c8181db04e88634764e7dc7d2b25116377662012a0611b8fddb76d" exitCode=0
Oct 30 23:41:14.294395 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:14.294298    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerDied","Data":"4bd45d2ac8c8181db04e88634764e7dc7d2b25116377662012a0611b8fddb76d"}
Oct 30 23:41:15.143273 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:15.143240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:15.143273 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:15.143274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:15.143273 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:15.143274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:15.143549 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:15.143362    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:15.143549 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:15.143441    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:15.143549 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:15.143526    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:16.300781 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:16.300748    2577 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="a634150600d67af9479bb8c5d3068cc04ca6eee83463ebb610dfe6e2019050ad" exitCode=0
Oct 30 23:41:16.301520 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:16.300831    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerDied","Data":"a634150600d67af9479bb8c5d3068cc04ca6eee83463ebb610dfe6e2019050ad"}
Oct 30 23:41:16.304683 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:16.304618    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" event={"ID":"f6b0becd-1e63-4dc2-a01e-3415794c4b46","Type":"ContainerStarted","Data":"cbbe4c1c29d7e2b27beac29f82439cdb07919a739b1ea7648343a529284d2414"}
Oct 30 23:41:16.304897 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:16.304883    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:16.319399 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:16.319381    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:16.630535 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:16.630493    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9" podStartSLOduration=6.699167617 podStartE2EDuration="21.630480258s" totalImagesPullingTime="14.931312641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:16.630279047 +0000 UTC m=+23.062015556" watchObservedRunningTime="2026-10-30 23:41:16.630480258 +0000 UTC m=+23.062216768"
Oct 30 23:41:17.143229 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:17.143203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:17.143422 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:17.143329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:17.143422 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:17.143343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:17.143422 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:17.143347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:17.143422 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:17.143399    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:17.143615 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:17.143456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:17.307108 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:17.307074    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:17.307506 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:17.307175    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:17.325752 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:17.325725    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:18.311464 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.311426    2577 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="dd95a02a0ea800524f71fc61589360aa3a70a1c9f24b4fbc7d2daa943a97ebd8" exitCode=0
Oct 30 23:41:18.311860 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.311505    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerDied","Data":"dd95a02a0ea800524f71fc61589360aa3a70a1c9f24b4fbc7d2daa943a97ebd8"}
Oct 30 23:41:18.773764 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.773574    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gl7bk"]
Oct 30 23:41:18.773925 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.773840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:18.773965 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:18.773922    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:18.776999 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.776973    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hrmp7"]
Oct 30 23:41:18.777122 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.777084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:18.777214 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:18.777194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:18.777584 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.777564    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t5xw7"]
Oct 30 23:41:18.777702 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:18.777658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:18.777758 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:18.777724    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:19.149472 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:19.149442    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:19.149656 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:19.149554    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:41:19.149962 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:19.149940    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:20.142981 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:20.142477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:20.142981 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:20.142651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:20.142981 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:20.142716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:20.142981 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:20.142828    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:21.143137 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:21.143113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:21.143479 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:21.143205    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:21.320240 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:21.320208    2577 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="1bf5afd2ce79862f2617e3ce2bb6c6b3f20d13f44ef6ed3107f736b4ac5cdc0f" exitCode=0
Oct 30 23:41:21.320367 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:21.320279    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerDied","Data":"1bf5afd2ce79862f2617e3ce2bb6c6b3f20d13f44ef6ed3107f736b4ac5cdc0f"}
Oct 30 23:41:22.143141 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:22.143108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:22.143141 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:22.143133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:22.143660 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:22.143204    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5xw7" podUID="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c"
Oct 30 23:41:22.143660 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:22.143332    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrmp7" podUID="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98"
Oct 30 23:41:22.324889 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:22.324819    2577 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="27ae4a377872128c3267642cc0b1ed689266a9536a354359834e20608f4e9e2e" exitCode=0
Oct 30 23:41:22.324889 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:22.324861    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerDied","Data":"27ae4a377872128c3267642cc0b1ed689266a9536a354359834e20608f4e9e2e"}
Oct 30 23:41:23.142580 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:23.142413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:23.142748 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:23.142663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl7bk" podUID="45dfef13-d5e6-4bff-98d2-47032ef6f624"
Oct 30 23:41:23.330226 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:23.330192    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" event={"ID":"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6","Type":"ContainerStarted","Data":"7d61d1b6c2ca4071d2f16477923e5aff535d4b7bfdb10aa232e093664ab440c5"}
Oct 30 23:41:23.537648 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:23.537598    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j7mw9" podStartSLOduration=19.263433314 podStartE2EDuration="28.537584185s" totalImagesPullingTime="9.274150871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:23.491288834 +0000 UTC m=+29.923025344" watchObservedRunningTime="2026-10-30 23:41:23.537584185 +0000 UTC m=+29.969320688"
Oct 30 23:41:23.925521 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:23.925498    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeReady"
Oct 30 23:41:23.925680 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:23.925605    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 23:41:24.018192 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.018159    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hx47s"]
Oct 30 23:41:24.039735 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.039710    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cm5bp"]
Oct 30 23:41:24.039846 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.039829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.058375 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.058353    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-45bcz"]
Oct 30 23:41:24.058480 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.058463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.073308 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.073291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.089104 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.089077    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:41:24.089104 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.089097    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:41:24.089316 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.089304    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6s2q2\""
Oct 30 23:41:24.098049 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.097726    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hx47s"]
Oct 30 23:41:24.098324 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.098306    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cm5bp"]
Oct 30 23:41:24.100396 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.100375    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45bcz"]
Oct 30 23:41:24.117817 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.117790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deaff92c-3bfd-4b69-a59d-3bf4d6716624-config-volume\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.117926 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.117835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.117971 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.117958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deaff92c-3bfd-4b69-a59d-3bf4d6716624-tmp-dir\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.118002 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.117981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrfr\" (UniqueName: \"kubernetes.io/projected/deaff92c-3bfd-4b69-a59d-3bf4d6716624-kube-api-access-8zrfr\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.143007 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.142980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:24.143007 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.143012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:24.190425 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190356    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:41:24.190425 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190362    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:41:24.190425 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190365    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:41:24.190658 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190428    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:41:24.190658 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190520    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:41:24.190658 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190539    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w5vpr\""
Oct 30 23:41:24.190658 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190557    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:41:24.190658 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190578    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvkr7\""
Oct 30 23:41:24.190658 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.190585    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:41:24.192289 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.192270    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:41:24.192397 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.192367    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\""
Oct 30 23:41:24.192474 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.192460    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:41:24.192521 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.192478    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\""
Oct 30 23:41:24.192571 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.192558    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:41:24.219119 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.219235 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95ec5cc3-2b66-46c0-9732-228496f671f4-cert\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.219235 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.219235 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfhn\" (UniqueName: \"kubernetes.io/projected/95ec5cc3-2b66-46c0-9732-228496f671f4-kube-api-access-4sfhn\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.219390 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ba94fb-766a-414b-a027-b378c42881e3-data-volume\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.219390 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.219263    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:24.219390 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deaff92c-3bfd-4b69-a59d-3bf4d6716624-tmp-dir\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.219390 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.219324    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.719304299 +0000 UTC m=+31.151040821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:24.219390 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrfr\" (UniqueName: \"kubernetes.io/projected/deaff92c-3bfd-4b69-a59d-3bf4d6716624-kube-api-access-8zrfr\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.219390 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.219606 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvd6\" (UniqueName: \"kubernetes.io/projected/69ba94fb-766a-414b-a027-b378c42881e3-kube-api-access-jdvd6\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.219606 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deaff92c-3bfd-4b69-a59d-3bf4d6716624-tmp-dir\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.219606 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deaff92c-3bfd-4b69-a59d-3bf4d6716624-config-volume\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.219606 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ba94fb-766a-414b-a027-b378c42881e3-crio-socket\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.219892 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.219876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deaff92c-3bfd-4b69-a59d-3bf4d6716624-config-volume\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.278943 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.278919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrfr\" (UniqueName: \"kubernetes.io/projected/deaff92c-3bfd-4b69-a59d-3bf4d6716624-kube-api-access-8zrfr\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.319978 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.319954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ba94fb-766a-414b-a027-b378c42881e3-crio-socket\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.320109 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.319998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95ec5cc3-2b66-46c0-9732-228496f671f4-cert\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.320109 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.320109 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfhn\" (UniqueName: \"kubernetes.io/projected/95ec5cc3-2b66-46c0-9732-228496f671f4-kube-api-access-4sfhn\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.320109 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ba94fb-766a-414b-a027-b378c42881e3-data-volume\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvd6\" (UniqueName: \"kubernetes.io/projected/69ba94fb-766a-414b-a027-b378c42881e3-kube-api-access-jdvd6\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.320158    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.320227    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.820208451 +0000 UTC m=+31.251944957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : configmap "kube-rbac-proxy" not found
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.320232    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.320288    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.82027403 +0000 UTC m=+31.252010540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:24.320304 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ba94fb-766a-414b-a027-b378c42881e3-crio-socket\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.320532 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.320419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ba94fb-766a-414b-a027-b378c42881e3-data-volume\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.322364 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.322342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95ec5cc3-2b66-46c0-9732-228496f671f4-cert\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.377914 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.377874    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvd6\" (UniqueName: \"kubernetes.io/projected/69ba94fb-766a-414b-a027-b378c42881e3-kube-api-access-jdvd6\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.378380 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.378185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfhn\" (UniqueName: \"kubernetes.io/projected/95ec5cc3-2b66-46c0-9732-228496f671f4-kube-api-access-4sfhn\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.447982 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.447924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:24.720549 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.720483    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"]
Oct 30 23:41:24.722947 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.722927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:24.723029 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.723018    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:24.723071 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.723063    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:25.723052936 +0000 UTC m=+32.154789425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:24.733777 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.733762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:24.777682 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.777651    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"]
Oct 30 23:41:24.782809 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.782788    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:41:24.782809 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.782805    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:41:24.782969 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.782921    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-28w6g\""
Oct 30 23:41:24.823320 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.823298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.823451 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.823340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:24.823451 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.823372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq26k\" (UniqueName: \"kubernetes.io/projected/0514cc38-b832-48da-9ed3-24db150b634f-kube-api-access-mq26k\") pod \"migrator-66df5d496b-vt56f\" (UID: \"0514cc38-b832-48da-9ed3-24db150b634f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:24.823451 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.823440    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:41:24.823596 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.823487    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:25.823474894 +0000 UTC m=+32.255211383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : configmap "kube-rbac-proxy" not found
Oct 30 23:41:24.823596 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.823504    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:24.823596 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:24.823569    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:25.823550816 +0000 UTC m=+32.255287320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:24.857897 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.857874    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45bcz"]
Oct 30 23:41:24.861921 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:41:24.861900    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ec5cc3_2b66_46c0_9732_228496f671f4.slice/crio-0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e WatchSource:0}: Error finding container 0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e: Status 404 returned error can't find the container with id 0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e
Oct 30 23:41:24.924323 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.924298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq26k\" (UniqueName: \"kubernetes.io/projected/0514cc38-b832-48da-9ed3-24db150b634f-kube-api-access-mq26k\") pod \"migrator-66df5d496b-vt56f\" (UID: \"0514cc38-b832-48da-9ed3-24db150b634f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:24.983852 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:24.983793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq26k\" (UniqueName: \"kubernetes.io/projected/0514cc38-b832-48da-9ed3-24db150b634f-kube-api-access-mq26k\") pod \"migrator-66df5d496b-vt56f\" (UID: \"0514cc38-b832-48da-9ed3-24db150b634f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:25.054643 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.054619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:25.143221 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.143194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:25.192862 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.192832    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:41:25.192985 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.192884    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\""
Oct 30 23:41:25.336406 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.336320    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45bcz" event={"ID":"95ec5cc3-2b66-46c0-9732-228496f671f4","Type":"ContainerStarted","Data":"0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e"}
Oct 30 23:41:25.429782 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.429752    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"]
Oct 30 23:41:25.432884 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:41:25.432859    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0514cc38_b832_48da_9ed3_24db150b634f.slice/crio-18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be WatchSource:0}: Error finding container 18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be: Status 404 returned error can't find the container with id 18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be
Oct 30 23:41:25.731483 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.731453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:25.731657 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:25.731592    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:25.731701 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:25.731664    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.731646762 +0000 UTC m=+34.163383269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:25.832576 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.832543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:25.832759 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:25.832600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:25.832759 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:25.832672    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:41:25.832759 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:25.832729    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.832715966 +0000 UTC m=+34.264452456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : configmap "kube-rbac-proxy" not found
Oct 30 23:41:25.832759 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:25.832739    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:25.832912 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:25.832789    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:27.832774336 +0000 UTC m=+34.264510841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:26.025055 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.024979    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2k2kl"]
Oct 30 23:41:26.043868 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.043845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.082580 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.082554    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2k2kl"]
Oct 30 23:41:26.093866 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.093847    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:41:26.094210 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.094191    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-znsw7\""
Oct 30 23:41:26.094364 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.094349    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:41:26.094434 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.094393    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:41:26.094624 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.094609    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:41:26.135248 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.135213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.135363 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.135285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2j2\" (UniqueName: \"kubernetes.io/projected/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-kube-api-access-ff2j2\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.135465 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.135448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-key\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.235853 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.235826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2j2\" (UniqueName: \"kubernetes.io/projected/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-kube-api-access-ff2j2\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.235953 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.235898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-key\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.235953 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.235935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.236611 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.236591    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.238803 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.238783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-key\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.293486 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.293429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2j2\" (UniqueName: \"kubernetes.io/projected/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-kube-api-access-ff2j2\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.338987 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.338962    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f" event={"ID":"0514cc38-b832-48da-9ed3-24db150b634f","Type":"ContainerStarted","Data":"18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be"}
Oct 30 23:41:26.372383 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.372358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:26.754711 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:26.754680    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2k2kl"]
Oct 30 23:41:26.757972 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:41:26.757943    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bf7af6_4d8c_4b3a_960f_56fbb356e7c5.slice/crio-85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb WatchSource:0}: Error finding container 85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb: Status 404 returned error can't find the container with id 85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb
Oct 30 23:41:27.043186 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.043113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:27.043186 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.043153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:27.043406 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.043253    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 23:41:27.043406 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.043318    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs podName:4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:59.043304512 +0000 UTC m=+65.475041002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs") pod "network-metrics-daemon-hrmp7" (UID: "4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98") : secret "metrics-daemon-secret" not found
Oct 30 23:41:27.045974 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.045954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:27.144103 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.144076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:27.146605 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.146585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:27.196330 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.196306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:27.225090 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.224955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7f6p_e895adf2-43c1-46f2-b2de-a3bddfd08f0c/konnectivity-agent/0.log" limit=1024
Oct 30 23:41:27.271790 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.271757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:27.343248 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.343216    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl" event={"ID":"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5","Type":"ContainerStarted","Data":"85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb"}
Oct 30 23:41:27.576427 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.576360    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t5xw7"]
Oct 30 23:41:27.580117 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:41:27.580084    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b67fb8d_7158_4441_9f05_5bd7aebdcf5c.slice/crio-d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af WatchSource:0}: Error finding container d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af: Status 404 returned error can't find the container with id d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af
Oct 30 23:41:27.684106 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.684076    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gl7bk"]
Oct 30 23:41:27.749530 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.749501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:27.749695 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.749676    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:27.749758 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.749748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.749731381 +0000 UTC m=+38.181467893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:27.810313 ip-10-0-131-123 kubenswrapper[2577]: W1030 23:41:27.810274    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dfef13_d5e6_4bff_98d2_47032ef6f624.slice/crio-904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac WatchSource:0}: Error finding container 904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac: Status 404 returned error can't find the container with id 904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac
Oct 30 23:41:27.851080 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.850841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:27.851080 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:27.850898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:27.851080 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.850938    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:41:27.851080 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.851001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.8509814 +0000 UTC m=+38.282717905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : configmap "kube-rbac-proxy" not found
Oct 30 23:41:27.851080 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.851010    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:27.851080 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:27.851061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:31.851047205 +0000 UTC m=+38.282783708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:28.347491 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:28.347457    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gl7bk" event={"ID":"45dfef13-d5e6-4bff-98d2-47032ef6f624","Type":"ContainerStarted","Data":"904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac"}
Oct 30 23:41:28.348796 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:28.348771    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t5xw7" event={"ID":"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c","Type":"ContainerStarted","Data":"d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af"}
Oct 30 23:41:28.350608 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:28.350494    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f" event={"ID":"0514cc38-b832-48da-9ed3-24db150b634f","Type":"ContainerStarted","Data":"6c27f11b27768200eaba01a8c92cc2f54adb3d9daa0a4b959ac59cffc14f265d"}
Oct 30 23:41:28.380992 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:28.380970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kmhdt_685220ca-ec2c-4955-91eb-53a4d5a9c62b/dns-node-resolver/0.log"
Oct 30 23:41:28.458476 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:28.458433    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f" podStartSLOduration=2.059036893 podStartE2EDuration="4.458418714s" totalImagesPullingTime="2.399381821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:28.458409267 +0000 UTC m=+34.890145784" watchObservedRunningTime="2026-10-30 23:41:28.458418714 +0000 UTC m=+34.890155225"
Oct 30 23:41:29.581906 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:29.581670    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ld74g_d645c1da-d44f-4765-bcdf-afe19821b63c/node-ca/0.log"
Oct 30 23:41:30.357871 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:30.357831    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45bcz" event={"ID":"95ec5cc3-2b66-46c0-9732-228496f671f4","Type":"ContainerStarted","Data":"e2454afe81ebd33eb280e9128613c5b8ab2dc75376de531195368672c227d8b8"}
Oct 30 23:41:30.781987 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:30.781917    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vt56f_0514cc38-b832-48da-9ed3-24db150b634f/migrator/0.log"
Oct 30 23:41:31.362207 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.362161    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl" event={"ID":"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5","Type":"ContainerStarted","Data":"1077eda2c75686d6f89730b5800666112c698a70e272e593e0f3c6ed921cc91d"}
Oct 30 23:41:31.471910 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.471857    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-45bcz" podStartSLOduration=2.509144412 podStartE2EDuration="7.471842302s" totalImagesPullingTime="4.96269789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:30.468174339 +0000 UTC m=+36.899910849" watchObservedRunningTime="2026-10-30 23:41:31.471842302 +0000 UTC m=+37.903578813"
Oct 30 23:41:31.472458 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.472429    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl" podStartSLOduration=2.178757622 podStartE2EDuration="6.472400125s" totalImagesPullingTime="4.293642503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:31.471747147 +0000 UTC m=+37.903483659" watchObservedRunningTime="2026-10-30 23:41:31.472400125 +0000 UTC m=+37.904136636"
Oct 30 23:41:31.783008 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.782931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:31.783404 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:31.783103    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:31.783404 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:31.783159    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:39.783146892 +0000 UTC m=+46.214883382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:31.884821 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.884234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:31.884821 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.884290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:31.884821 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:31.884394    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:31.884821 ip-10-0-131-123 kubenswrapper[2577]: E1030 23:41:31.884455    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:39.884437269 +0000 UTC m=+46.316173773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:31.884821 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:31.884810    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:33.368664 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:33.368612    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t5xw7" event={"ID":"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c","Type":"ContainerStarted","Data":"af28eb3ae3c459cb24017a915282a601e6c18d730ba9150ef99fd75c0516b839"}
Oct 30 23:41:33.369131 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:33.368787    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:33.477214 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:33.477151    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t5xw7" podStartSLOduration=33.405166146 podStartE2EDuration="38.47713388s" totalImagesPullingTime="5.071967734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:33.477062126 +0000 UTC m=+39.908798636" watchObservedRunningTime="2026-10-30 23:41:33.47713388 +0000 UTC m=+39.908870391"
Oct 30 23:41:34.630778 ip-10-0-131-123 kubenswrapper[2577]: I1030 23:41:34.630649    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:41:34.630829 ip-10-0-131-123 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 23:41:34.650098 ip-10-0-131-123 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 23:41:34.650265 ip-10-0-131-123 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 23:41:34.650426 ip-10-0-131-123 systemd[1]: kubelet.service: Consumed 2.319s CPU time.
Oct 30 23:41:34.665212 ip-10-0-131-123 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:41:34.738576 ip-10-0-131-123 kubenswrapper[4466]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:41:34.738576 ip-10-0-131-123 kubenswrapper[4466]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:41:34.738576 ip-10-0-131-123 kubenswrapper[4466]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:41:34.738576 ip-10-0-131-123 kubenswrapper[4466]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:41:34.740520 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740445    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.740520 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740519    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.740520 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740522    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740527    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740531    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740534    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740537    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740540    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740543    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740545    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740548    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740550    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740553    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740556    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740558    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740561    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740563    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740566    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740569    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740571    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740574    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740576    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.740619 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740579    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740581    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740583    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740586    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740589    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740599    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740602    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740605    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740608    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740610    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740613    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740615    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740618    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740621    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740623    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740626    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740644    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740649    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740653    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.741129 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740657    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740662    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740665    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740668    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740670    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740673    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740675    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740678    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740681    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740683    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740686    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740688    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740691    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740694    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740696    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740698    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740701    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740704    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740706    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740709    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.741572 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740712    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740715    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740717    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740722    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740726    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740731    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740734    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740737    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740739    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740742    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740744    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740747    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740749    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740752    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740755    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740757    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740760    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740763    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740765    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.742050 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740768    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740770    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740773    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740775    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740778    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740780    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740783    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740785    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740787    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740790    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740792    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740795    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740797    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740800    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740804    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740807    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740809    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740811    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740815    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740818    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.742509 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740822    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740825    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740827    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740829    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740832    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.740835    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741556    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741563    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741566    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741569    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741571    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741574    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741577    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741580    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741583    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741585    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741588    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741590    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741593    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.742997 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741596    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741598    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741601    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741603    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741605    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741608    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741610    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741614    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741617    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741619    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741622    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741625    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741628    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741646    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741651    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741654    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741657    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741659    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741662    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.743449 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741665    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741667    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741669    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741673    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741677    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741680    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741683    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741685    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741688    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741690    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741693    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741695    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741698    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741700    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741702    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741706    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741709    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741712    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741714    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.743947 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741717    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741720    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741724    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741727    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741729    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741732    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741735    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741738    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741740    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741742    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741745    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741748    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741750    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741753    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741755    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741762    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741765    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741768    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741770    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741773    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.744412 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741775    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741778    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741780    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741783    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741786    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741789    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741791    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741794    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741796    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741798    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741801    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741803    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741806    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741808    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741811    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741814    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741817    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741820    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741822    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741824    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.744901 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741827    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741829    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741832    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741834    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741837    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741839    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741842    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741845    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741847    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741850    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741852    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741856    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741858    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741861    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.741863    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742317    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742322    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742325    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742328    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742330    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.745384 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742333    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742336    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742338    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742341    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742344    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742347    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742350    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742353    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742356    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742359    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742361    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742364    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742366    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742369    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742372    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742374    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742377    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742379    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742382    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.745865 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742385    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742387    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742390    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742392    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742395    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742397    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742399    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742402    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742405    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742409    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742414    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742419    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742425    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742428    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742430    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742433    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742436    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742438    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742440    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.746327 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742443    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742445    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742448    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742451    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742453    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742455    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742458    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742460    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742463    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742465    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742468    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742471    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742473    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742475    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742478    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742480    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742483    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742486    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742488    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.746960 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742490    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742493    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742495    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742497    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742500    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742502    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742507    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742510    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742513    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742516    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742519    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742522    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742524    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742526    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742529    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742531    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742534    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742537    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742539    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742542    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.747765 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742544    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742547    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742549    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742552    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742554    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742557    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742559    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742562    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742564    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742567    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742570    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742572    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742575    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742577    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742580    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742582    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742585    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742587    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742591    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.748620 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742593    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742596    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742599    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742601    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.742603    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742711    4466 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742721    4466 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742728    4466 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742735    4466 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742739    4466 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742743    4466 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742747    4466 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742752    4466 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742755    4466 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742758    4466 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742761    4466 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742765    4466 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742768    4466 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742771    4466 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742773    4466 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742776    4466 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742779    4466 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742782    4466 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:41:34.749408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742786    4466 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742789    4466 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742792    4466 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742795    4466 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742798    4466 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742802    4466 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742805    4466 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742808    4466 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742814    4466 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742817    4466 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742820    4466 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742823    4466 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742826    4466 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742829    4466 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742833    4466 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742836    4466 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742839    4466 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742841    4466 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742844    4466 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742847    4466 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742851    4466 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742854    4466 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742858    4466 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742861    4466 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:41:34.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742864    4466 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742868    4466 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742871    4466 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742874    4466 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742877    4466 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742880    4466 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742883    4466 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742886    4466 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742889    4466 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742892    4466 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742894    4466 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742897    4466 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742901    4466 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742904    4466 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742907    4466 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742911    4466 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742917    4466 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742920    4466 flags.go:64] FLAG: --help="false"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742923    4466 flags.go:64] FLAG: --hostname-override="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742926    4466 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742929    4466 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742932    4466 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742935    4466 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:41:34.751086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742938    4466 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742941    4466 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742944    4466 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742947    4466 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742950    4466 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742953    4466 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742957    4466 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742960    4466 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742963    4466 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742965    4466 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742968    4466 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742971    4466 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742974    4466 flags.go:64] FLAG: --lock-file=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742977    4466 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742979    4466 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742982    4466 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742988    4466 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742991    4466 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742994    4466 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742996    4466 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.742999    4466 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743003    4466 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743005    4466 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743008    4466 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743012    4466 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:41:34.751621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743019    4466 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743023    4466 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743025    4466 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743030    4466 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743034    4466 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743039    4466 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743043    4466 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743048    4466 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743052    4466 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743062    4466 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743065    4466 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743069    4466 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743071    4466 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743074    4466 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743078    4466 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743081    4466 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743084    4466 flags.go:64] FLAG: --port="10250"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743086    4466 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743089    4466 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-09a18c6d6c5eaa424"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743093    4466 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743095    4466 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743098    4466 flags.go:64] FLAG: --register-node="true"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743101    4466 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743105    4466 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743108    4466 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743110    4466 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:41:34.752265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743113    4466 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743117    4466 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743120    4466 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743123    4466 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743126    4466 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743128    4466 flags.go:64] FLAG: --runonce="false"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743134    4466 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743137    4466 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743140    4466 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743143    4466 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743146    4466 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743149    4466 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743153    4466 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743156    4466 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743159    4466 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743162    4466 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743165    4466 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743167    4466 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743170    4466 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743173    4466 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743176    4466 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743182    4466 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743185    4466 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743187    4466 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743191    4466 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:41:34.752891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743194    4466 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743197    4466 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743199    4466 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743202    4466 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743205    4466 flags.go:64] FLAG: --v="2"
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743209    4466 flags.go:64] FLAG: --version="false"
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743213    4466 flags.go:64] FLAG: --vmodule=""
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743217    4466 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743220    4466 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743366    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743370    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743373    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743376    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743382    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743385    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743387    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743390    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743392    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743395    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743397    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743400    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743402    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.753480 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743405    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743408    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743410    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743412    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743415    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743418    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743421    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743423    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743426    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743428    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743431    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743433    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743436    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743438    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743440    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743443    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743446    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743448    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743451    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.754070 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743453    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743456    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743459    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743462    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743467    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743470    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743472    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743475    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743477    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743480    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743482    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743485    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743488    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743490    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743493    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743495    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743498    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743500    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743503    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743506    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.754523 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743508    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743511    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743514    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743516    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743519    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743522    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743524    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743526    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743529    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743532    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743534    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743536    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743539    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743541    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743544    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743547    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743553    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743556    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743559    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743561    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.755025 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743564    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743566    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743571    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743575    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743578    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743581    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743584    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743587    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743590    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743592    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743595    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743597    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743600    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743602    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743605    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743608    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743610    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743613    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743615    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.755753 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743619    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743622    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743625    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743628    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743642    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743645    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743648    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743651    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743653    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743658    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743661    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743664    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743666    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743669    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743671    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.743698    4466 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:41:34.756446 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743746    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743750    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743753    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743756    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743759    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743761    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743764    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743766    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743769    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743771    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743774    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743777    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743780    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743782    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743785    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743788    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743790    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743793    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743795    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743798    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.757003 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743800    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743803    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743805    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743808    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743810    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743814    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743818    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743820    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743823    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743825    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743828    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743830    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743833    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743835    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743838    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743841    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743843    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743846    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743848    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.757515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743851    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743853    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743855    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743858    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743860    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743863    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743866    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743868    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743871    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743874    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743876    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743878    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743881    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743883    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743886    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743888    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743891    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743894    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743899    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.757979 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743903    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743906    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743908    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743911    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743914    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743917    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743919    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743922    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743924    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743927    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743929    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743932    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743934    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743937    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743939    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743943    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743946    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743949    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743951    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.758515 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743954    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743957    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743960    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743963    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743965    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743967    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743970    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743972    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743975    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743977    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743979    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743982    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743986    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743989    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743992    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743994    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.743997    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744000    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744002    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744005    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.759151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744007    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744010    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744013    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744015    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744017    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744020    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744022    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744025    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.744027    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.750945    4466 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.750964    4466 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.750988    4466 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.750995    4466 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751078    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751085    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751090    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751095    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751098    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751101    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751104    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.759892 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751107    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751110    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751113    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751115    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751118    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751120    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751123    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751126    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751129    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751131    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751134    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751137    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751139    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751142    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751144    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751147    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751150    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751152    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751155    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751157    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.760399 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751160    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751162    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751165    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751168    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751171    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751173    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751176    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751180    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751184    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751188    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751190    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751193    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751195    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751198    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751200    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751203    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751206    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751208    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751211    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.760882 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751214    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751217    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751219    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751223    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751225    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751228    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751231    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751234    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751237    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751239    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751242    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751245    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751247    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751250    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751267    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751272    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751275    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751278    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751281    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751283    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.761354 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751286    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751289    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751291    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751294    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751296    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751299    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751301    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751304    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751306    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751309    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751311    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751314    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751316    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751320    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751324    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751327    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751330    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751333    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751336    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.761962 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751338    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751341    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751344    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751346    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751349    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751351    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751354    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751356    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751359    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751362    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751364    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751367    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751370    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751372    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751375    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751377    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751380    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751382    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751385    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.762583 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751387    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751389    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.751417    4466 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751575    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751580    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751585    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751589    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751591    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751594    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751597    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751600    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751603    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751605    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751609    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751613    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.763089 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751616    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751619    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751622    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751625    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751628    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751647    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751652    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751657    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751661    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751666    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751669    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751672    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751675    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751677    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751680    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751682    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751685    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751687    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751690    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.763451 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751692    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751694    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751697    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751699    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751702    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751704    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751706    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751710    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751713    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751715    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751718    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751720    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751723    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751726    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751728    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751730    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751733    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751735    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751738    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751740    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.763968 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751743    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751745    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751748    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751750    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751752    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751755    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751757    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751760    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751763    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751765    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751768    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751770    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751773    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751775    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751777    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751780    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751783    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751785    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751788    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.764421 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751790    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751794    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751796    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751799    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751801    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751804    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751807    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751809    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751812    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751814    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751817    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751819    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751822    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751824    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751826    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751829    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751832    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751834    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751836    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751839    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.764878 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751842    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751844    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751847    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751849    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751852    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751854    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751857    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751859    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751861    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751864    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751866    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751869    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751871    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751874    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751876    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.765356 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751879    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.751902    4466 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751939    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751944    4466 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751947    4466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751950    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751952    4466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751955    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751958    4466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751961    4466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751963    4466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751966    4466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751968    4466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751971    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751973    4466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:34.765718 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751976    4466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751978    4466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751981    4466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751983    4466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751986    4466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751988    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751992    4466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751995    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.751998    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752001    4466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752004    4466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752006    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752009    4466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752011    4466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752014    4466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752018    4466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752021    4466 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752025    4466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:34.766104 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752028    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752031    4466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752034    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752037    4466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752040    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752043    4466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752045    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752048    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752050    4466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752053    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752056    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752058    4466 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752060    4466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752063    4466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752065    4466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752068    4466 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752070    4466 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752073    4466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752075    4466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752078    4466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:34.766875 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752080    4466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752083    4466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752085    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752088    4466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752090    4466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752093    4466 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752096    4466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752098    4466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752101    4466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752104    4466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752106    4466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752109    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752112    4466 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752114    4466 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752117    4466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752119    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752122    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752124    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752127    4466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752129    4466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:34.767578 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752132    4466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752134    4466 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752137    4466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752139    4466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752141    4466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752144    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752147    4466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752149    4466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752151    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752154    4466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752156    4466 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752159    4466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752161    4466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752164    4466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752166    4466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752169    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752171    4466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752174    4466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752176    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:34.768231 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752179    4466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752182    4466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752185    4466 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752187    4466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752190    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752192    4466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752195    4466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752200    4466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752203    4466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752205    4466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752208    4466 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752210    4466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752213    4466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752215    4466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752219    4466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:34.752222    4466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.752276    4466 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753202    4466 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753258    4466 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753431    4466 server.go:1014] "Starting client certificate rotation"
Oct 30 23:41:34.768888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753564    4466 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753612    4466 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:35:54 +0000 UTC" deadline="2028-07-03 01:49:19.278976114 +0000 UTC"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753653    4466 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14666h7m44.525325008s"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.753885    4466 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.755046    4466 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.756656    4466 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.758213    4466 log.go:25] "Validated CRI v1 image API"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.759033    4466 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:41:34.769687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.760526    4466 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a09f4bc-237f-4cc4-b12d-89d266cd1d40:/dev/nvme0n1p3 9132e87a-039d-44cd-9453-9b15e1e01ab3:/dev/nvme0n1p4]
Oct 30 23:41:34.769998 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.760558    4466 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3/userdata/shm major:0 minor:145 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0514cc38-b832-48da-9ed3-24db150b634f/volumes/kubernetes.io~projected/kube-api-access-mq26k:{mountpoint:/var/lib/kubelet/pods/0514cc38-b832-48da-9ed3-24db150b634f/volumes/kubernetes.io~projected/kube-api-access-mq26k major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24273fe9-3b2a-421c-bec7-26c6af951dcb/volumes/kubernetes.io~projected/kube-api-access-7bg7r:{mountpoint:/var/lib/kubelet/pods/24273fe9-3b2a-421c-bec7-26c6af951dcb/volumes/kubernetes.io~projected/kube-api-access-7bg7r major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c/volumes/kubernetes.io~projected/kube-api-access-zgxnf:{mountpoint:/var/lib/kubelet/pods/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c/volumes/kubernetes.io~projected/kube-api-access-zgxnf major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6/volumes/kubernetes.io~projected/kube-api-access-jjdcp:{mountpoint:/var/lib/kubelet/pods/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6/volumes/kubernetes.io~projected/kube-api-access-jjdcp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45dfef13-d5e6-4bff-98d2-47032ef6f624/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/45dfef13-d5e6-4bff-98d2-47032ef6f624/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98/volumes/kubernetes.io~projected/kube-api-access-fjvvz:{mountpoint:/var/lib/kubelet/pods/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98/volumes/kubernetes.io~projected/kube-api-access-fjvvz major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b/volumes/kubernetes.io~projected/kube-api-access-2grks:{mountpoint:/var/lib/kubelet/pods/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b/volumes/kubernetes.io~projected/kube-api-access-2grks major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/685220ca-ec2c-4955-91eb-53a4d5a9c62b/volumes/kubernetes.io~projected/kube-api-access-pqsst:{mountpoint:/var/lib/kubelet/pods/685220ca-ec2c-4955-91eb-53a4d5a9c62b/volumes/kubernetes.io~projected/kube-api-access-pqsst major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ba94fb-766a-414b-a027-b378c42881e3/volumes/kubernetes.io~projected/kube-api-access-jdvd6:{mountpoint:/var/lib/kubelet/pods/69ba94fb-766a-414b-a027-b378c42881e3/volumes/kubernetes.io~projected/kube-api-access-jdvd6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95ec5cc3-2b66-46c0-9732-228496f671f4/volumes/kubernetes.io~projected/kube-api-access-4sfhn:{mountpoint:/var/lib/kubelet/pods/95ec5cc3-2b66-46c0-9732-228496f671f4/volumes/kubernetes.io~projected/kube-api-access-4sfhn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95ec5cc3-2b66-46c0-9732-228496f671f4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/95ec5cc3-2b66-46c0-9732-228496f671f4/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97c08a18-b069-47a5-bdfd-7fca0c7767ae/volumes/kubernetes.io~projected/kube-api-access-tlm89:{mountpoint:/var/lib/kubelet/pods/97c08a18-b069-47a5-bdfd-7fca0c7767ae/volumes/kubernetes.io~projected/kube-api-access-tlm89 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~projected/kube-api-access-wd9h7:{mountpoint:/var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~projected/kube-api-access-wd9h7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d645c1da-d44f-4765-bcdf-afe19821b63c/volumes/kubernetes.io~projected/kube-api-access-qqds5:{mountpoint:/var/lib/kubelet/pods/d645c1da-d44f-4765-bcdf-afe19821b63c/volumes/kubernetes.io~projected/kube-api-access-qqds5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5/volumes/kubernetes.io~projected/kube-api-access-ff2j2:{mountpoint:/var/lib/kubelet/pods/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5/volumes/kubernetes.io~projected/kube-api-access-ff2j2 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5/volumes/kubernetes.io~secret/signing-key major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deaff92c-3bfd-4b69-a59d-3bf4d6716624/volumes/kubernetes.io~projected/kube-api-access-8zrfr:{mountpoint:/var/lib/kubelet/pods/deaff92c-3bfd-4b69-a59d-3bf4d6716624/volumes/kubernetes.io~projected/kube-api-access-8zrfr major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e895adf2-43c1-46f2-b2de-a3bddfd08f0c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e895adf2-43c1-46f2-b2de-a3bddfd08f0c/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volumes/kubernetes.io~projected/kube-api-access-68qrj:{mountpoint:/var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volumes/kubernetes.io~projected/kube-api-access-68qrj major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/50459909cd53e65af429f1acbad6c685e6662c0a2f70a698bb93fed786f7d567/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/b6b2bbe05c496ffdd087fcd4042c3f2931250654d569616b06e13b74b3d39a1f/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f609d9a0ed5dbde900d4cf8dfdada34d36267be99f5b16d88e270c596dfc13cf/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/bd4c6400fbd543b1b57dd2b3ca303a977c6d7a6ef9b27bb25c2f5e12e01739df/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/var/lib/containers/storage/overlay/d80b0f438dfd04ce867376cbc3a6f4258bc5a8ce5233c717ff09cd66dbf2673c/merged major:0 minor:117 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/77bf42ddc4d5ca63d44a77c902fe8becf18fcf4a87572781925bc68de683867f/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-125:{mountpoint:/var/lib/containers/storage/overlay/8d42c37462117f45c0a3c2d882ece4f2e4052ce357b3ebfa052522286e867e4b/merged major:0 minor:125 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/d45544157ecb02cb750b2daa7cd32e3ca471e89f32694b833653094f7291b260/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/0bb3e1248d5ba1d89fbe3ba0350d7e7fb6e588f3d256e4da513638488308a67e/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/97ad23755e7316abf01a25ab2fca76ae4bd1b32795a8efce0414bdddd9736343/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/8bbbafb0b975cc2552ec53a5eab4b9aa37b44481e635f41654f8100d31906f63/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/db7fabb8160b4ab4bce2c1ca2ead04bcf8ec5f9f7b29ad8b8832dbb558fc0621/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7a9b97e3b2bd78de60030bcffae56281eabfd3b7258f02a20bd220cc2421bffa/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/93e19f9a71f957abddef79dea962d56a2b3e0e89fdfc850aaadf379b5db110b7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/966a5a2d6794d5c88376bf25c70447e0a3fdd04c742c2fedc056c070456b96d2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/24c8dba4a68472474bf43ab150ee24fea620c1f05b1f88bdfa734960edcfe73f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/33b1c3781599559a6315c75d92454ce064d54a56fba8cb3b3358ceda11173fef/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/67119354afd4927c90fb5797946e0c447b66730d501f425b36af682039a83f1b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/08d0ae15e9a4b6c43048b80fd27759db0356151eb2d6695117b25fbe841b444d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/c7fa1f304cbbcc1c6d66fe5fca116f97ec729a447c155658e8dddbf1d7c05978/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/96f51ffd190559875ad58e2570632d968bb86991ea495999fdec35a8bdba98df/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/52b910493d75c2c17abfbd0c2b5d8c106aa8395dd5a01fddc99773266ca94f6e/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/f655c30731961bd53498b52765b4885dfadda26b86cd62bf0b173b4084037e71/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/e9010f60f9d1d85a175564f8b35a454564d0cdfbbd68fd42b13b9f937e89b3a2/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/f39ca6281ec0883b15105e45f2b85b386e6ca034eb715df4ed174ed650ebf258/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/0716ea43968e0f57afb182b7ad1670dafbb9c3ab4184511e876dd2cf65526393/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d8769cb59aac191926ab3dd545e6bd9cb51462630da0778a437b438c7e41c7c7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/dfd018481bc49ff9ecec6f85920423fb97f0fd48736ab4b3023120ef0947dfea/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/80e6d6f1f5122bba77644088eecb4604f41b2df9cd6841ec813c96031800b29e/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7957e86e753b4cfa369035f2715a42cd83d30f6d4e790a161b38201274a47d30/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/491778c424c6ca6c3eb5073d8b99001e2e8fa3ad8981699c773d6c938b8618ce/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/feff762558f32d9a8bc048cfea279d3d7a538d91d8f3df7417eca476c474dc99/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ea1215bffb75335da2c2bde87045fb0f644ae85236dba30c19c00eca4402ccc6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6683fbddbda14588d514db26c1bfec700247ff8bab5b8241a422ba9e4305c344/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0e889e7324bab12ef5bfb5fe3aac9986ddb01f9a596c3c70e10e0a62a7087c6e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/abedb6711d9b652b991678f8c0f1f0e2895a9e4bc2c937938f4f745e42bbfd82/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/65c5dcb1baf60749d7220f1f21e3ef1b7cf166e65e75126b94a51141fd8f9119/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/768a5b064109d8941bd4d6eb6381c0e7ffa39171bc257a852b975afa57f12e89/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2524c429e9457edea6591f47d10a2c9e07eb1d3f43642f757adc97712a3e2568/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/80dbefb63d335a3dbaea3f6668bf3622f40d7b71ee824d5cb6ee7302b755c7ca/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/7f67076d897c577e8ccf58b02044026dc5e6c498d4cd7b8cac22c346367bcbd7/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.768101    4466 manager.go:217] Machine: {Timestamp:2026-10-30 23:41:34.767291273 +0000 UTC m=+0.057839589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201448 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c979b3759a4f8432d33934e679e75 SystemUUID:ec2c979b-3759-a4f8-432d-33934e679e75 BootID:aa6428a3-06f4-4cda-ab80-5db31089f211 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31a9b1f37d4cd9bbec482b1f1d2882957e2d258fd20cc31663918d510c255696/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-125 DeviceMajor:0 DeviceMinor:125 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deaff92c-3bfd-4b69-a59d-3bf4d6716624/volumes/kubernetes.io~projected/kube-api-access-8zrfr DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45dfef13-d5e6-4bff-98d2-47032ef6f624/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/904e6dcc72952e4dcbd2d982220feef2d7f904c8ea00cd5082e95edcc43e09ac/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~projected/kube-api-access-wd9h7 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae6abb22474224f273f276b3656f0ef39feb9123fa0d48b696158ddc53db0af3/userdata/shm DeviceMajor:0 DeviceMinor:145 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95ec5cc3-2b66-46c0-9732-228496f671f4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18714991c1c2872f3e02115cc57d05cbf01db0c6a835063e43eb43a22e7146be/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e895adf2-43c1-46f2-b2de-a3bddfd08f0c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d645c1da-d44f-4765-bcdf-afe19821b63c/volumes/kubernetes.io~projected/kube-api-access-qqds5 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2f8e4d5c8719aec9e3c5748c76742a2467268b1de140c0b304c54c53430c527/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c69513ef610abeda953356b021acbcce35120c9ef2da5df4ac718d227156b1a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d74d189d92b739a71aa7b0f0f57013a6c53fd3f81df6efd599f041b55609b4af/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6/volumes/kubernetes.io~projected/kube-api-access-jjdcp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/159b3742f1379168fa1413a47d23f967cc8a6ed6cfd407ae54225f098e24bac9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5/volumes/kubernetes.io~projected/kube-api-access-ff2j2 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97c08a18-b069-47a5-bdfd-7fca0c7767ae/volumes/kubernetes.io~projected/kube-api-access-tlm89 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-117 DeviceMajor:0 DeviceMinor:117 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85e3b3a6a2ad01f50c66d6be278df9a9d5d6abf69e65cf08e919f671913b8dbb/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c/volumes/kubernetes.io~projected/kube-api-access-zgxnf DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/685220ca-ec2c-4955-91eb-53a4d5a9c62b/volumes/kubernetes.io~projected/kube-api-access-pqsst DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98/volumes/kubernetes.io~projected/kube-api-access-fjvvz DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0532fc726497bbb3a9580619059c5b790d8c4b7dd6c17665b110f401d696510e/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24273fe9-3b2a-421c-bec7-26c6af951dcb/volumes/kubernetes.io~projected/kube-api-access-7bg7r DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f8297f-f076-43cb-ba9f-93a455101c9f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6b0becd-1e63-4dc2-a01e-3415794c4b46/volumes/kubernetes.io~projected/kube-api-access-68qrj DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b/volumes/kubernetes.io~projected/kube-api-access-2grks DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f4117de419a2bc64e88de12f06dc7d373ea666b06e4ff8c06ade914a06bc7fb/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ab68bb6586474f935a56176cee760422556f2944011f288a24a234eff51ed82/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0514cc38-b832-48da-9ed3-24db150b634f/volumes/kubernetes.io~projected/kube-api-access-mq26k DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df4ee850c98de70e4cdf54aac45a2a70ffff8807cf52380022c7261700040208/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69ba94fb-766a-414b-a027-b378c42881e3/volumes/kubernetes.io~projected/kube-api-access-jdvd6 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/95ec5cc3-2b66-46c0-9732-228496f671f4/volumes/kubernetes.io~projected/kube-api-access-4sfhn DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0532fc726497bbb MacAddress:76:b5:42:0b:5f:e2 Speed:10000 Mtu:8901} {Name:18714991c1c2872 MacAddress:de:f3:18:73:15:6f Speed:10000 Mtu:8901} {Name:85e3b3a6a2ad01f MacAddress:f2:8f:0b:05:e9:61 Speed:10000 Mtu:8901} {Name:904e6dcc72952e4 MacAddress:9e:27:18:9c:14:3d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c2:d8:cc:e7 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:58:38:90:b5:97 Speed:0 Mtu:8901} {Name:d74d189d92b739a MacAddress:86:c7:51:0f:78:dd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c2:d8:cc:e7 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:72:42:a7:af:95:c7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:2f:e5:ac:db:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.768385    4466 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.768472    4466 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.768821    4466 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.768848    4466 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-123.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769024    4466 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769035    4466 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769050    4466 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769075    4466 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769175    4466 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769380    4466 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769507    4466 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769550    4466 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769567    4466 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769580    4466 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769592    4466 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:41:34.770249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.769603    4466 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:41:34.771334 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.770342    4466 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:41:34.771334 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.770355    4466 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:41:34.771334 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.770842    4466 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:41:34.772873 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.772855    4466 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:41:34.772873 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.772869    4466 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:41:34.772999 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.772976    4466 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 23:41:34.773191 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773175    4466 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:41:34.773262 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773214    4466 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:41:34.773333 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773322    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:41:34.773383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773342    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:41:34.773383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773352    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:41:34.773383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773360    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:41:34.773383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773369    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:41:34.773383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773378    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773386    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773395    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773406    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773414    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773426    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773440    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773480    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:41:34.773617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.773490    4466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:41:34.777033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.777019    4466 server.go:1263] "Started kubelet"
Oct 30 23:41:34.777278 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.777232    4466 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:41:34.777363 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.777297    4466 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:41:34.777439 ip-10-0-131-123 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:41:34.777616 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.777605    4466 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:41:34.777682 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.777524    4466 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:41:34.778953 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.778934    4466 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:41:34.786422 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.786402    4466 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:41:34.786784 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.786760    4466 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:41:34.786901 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.786870    4466 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:35:55 +0000 UTC" deadline="2028-05-22 11:31:40.104043983 +0000 UTC"
Oct 30 23:41:34.786969 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.786901    4466 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13667h50m5.31714569s"
Oct 30 23:41:34.787019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.786993    4466 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:41:34.787019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.787005    4466 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:41:34.787169 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.787137    4466 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found"
Oct 30 23:41:34.787401 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.787386    4466 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.788991    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d645c1da-d44f-4765-bcdf-afe19821b63c" volumeName="kubernetes.io/projected/d645c1da-d44f-4765-bcdf-afe19821b63c-kube-api-access-qqds5" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789042    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5" volumeName="kubernetes.io/projected/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-kube-api-access-ff2j2" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789055    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e895adf2-43c1-46f2-b2de-a3bddfd08f0c" volumeName="kubernetes.io/configmap/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-konnectivity-ca" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789068    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b0becd-1e63-4dc2-a01e-3415794c4b46" volumeName="kubernetes.io/secret/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789080    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b67fb8d-7158-4441-9f05-5bd7aebdcf5c" volumeName="kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789091    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" volumeName="kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789102    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45dfef13-d5e6-4bff-98d2-47032ef6f624" volumeName="kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789115    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ba94fb-766a-414b-a027-b378c42881e3" volumeName="kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789130    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ba94fb-766a-414b-a027-b378c42881e3" volumeName="kubernetes.io/projected/69ba94fb-766a-414b-a027-b378c42881e3-kube-api-access-jdvd6" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789143    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24273fe9-3b2a-421c-bec7-26c6af951dcb" volumeName="kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789154    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24273fe9-3b2a-421c-bec7-26c6af951dcb" volumeName="kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-daemon-config" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789164    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" volumeName="kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789175    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b" volumeName="kubernetes.io/projected/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kube-api-access-2grks" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789188    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="685220ca-ec2c-4955-91eb-53a4d5a9c62b" volumeName="kubernetes.io/projected/685220ca-ec2c-4955-91eb-53a4d5a9c62b-kube-api-access-pqsst" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789203    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ba94fb-766a-414b-a027-b378c42881e3" volumeName="kubernetes.io/empty-dir/69ba94fb-766a-414b-a027-b378c42881e3-data-volume" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789216    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" volumeName="kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789224    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ec5cc3-2b66-46c0-9732-228496f671f4" volumeName="kubernetes.io/projected/95ec5cc3-2b66-46c0-9732-228496f671f4-kube-api-access-4sfhn" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789231    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f8297f-f076-43cb-ba9f-93a455101c9f" volumeName="kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-tuned" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789238    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d645c1da-d44f-4765-bcdf-afe19821b63c" volumeName="kubernetes.io/configmap/d645c1da-d44f-4765-bcdf-afe19821b63c-serviceca" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789246    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b0becd-1e63-4dc2-a01e-3415794c4b46" volumeName="kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-config" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789257    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24273fe9-3b2a-421c-bec7-26c6af951dcb" volumeName="kubernetes.io/projected/24273fe9-3b2a-421c-bec7-26c6af951dcb-kube-api-access-7bg7r" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789269    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" volumeName="kubernetes.io/projected/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-kube-api-access-jjdcp" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789280    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98" volumeName="kubernetes.io/projected/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-kube-api-access-fjvvz" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789291    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f8297f-f076-43cb-ba9f-93a455101c9f" volumeName="kubernetes.io/projected/a7f8297f-f076-43cb-ba9f-93a455101c9f-kube-api-access-wd9h7" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789303    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5" volumeName="kubernetes.io/configmap/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-cabundle" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789315    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b0becd-1e63-4dc2-a01e-3415794c4b46" volumeName="kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789325    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ec5cc3-2b66-46c0-9732-228496f671f4" volumeName="kubernetes.io/secret/95ec5cc3-2b66-46c0-9732-228496f671f4-cert" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789336    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaff92c-3bfd-4b69-a59d-3bf4d6716624" volumeName="kubernetes.io/configmap/deaff92c-3bfd-4b69-a59d-3bf4d6716624-config-volume" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789351    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e895adf2-43c1-46f2-b2de-a3bddfd08f0c" volumeName="kubernetes.io/secret/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-agent-certs" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789360    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b0becd-1e63-4dc2-a01e-3415794c4b46" volumeName="kubernetes.io/projected/f6b0becd-1e63-4dc2-a01e-3415794c4b46-kube-api-access-68qrj" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789372    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="685220ca-ec2c-4955-91eb-53a4d5a9c62b" volumeName="kubernetes.io/empty-dir/685220ca-ec2c-4955-91eb-53a4d5a9c62b-tmp-dir" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789384    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97c08a18-b069-47a5-bdfd-7fca0c7767ae" volumeName="kubernetes.io/projected/97c08a18-b069-47a5-bdfd-7fca0c7767ae-kube-api-access-tlm89" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789395    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaff92c-3bfd-4b69-a59d-3bf4d6716624" volumeName="kubernetes.io/projected/deaff92c-3bfd-4b69-a59d-3bf4d6716624-kube-api-access-8zrfr" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789407    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f8297f-f076-43cb-ba9f-93a455101c9f" volumeName="kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-tmp" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789417    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5" volumeName="kubernetes.io/secret/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-key" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789428    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaff92c-3bfd-4b69-a59d-3bf4d6716624" volumeName="kubernetes.io/empty-dir/deaff92c-3bfd-4b69-a59d-3bf4d6716624-tmp-dir" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789438    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6b0becd-1e63-4dc2-a01e-3415794c4b46" volumeName="kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-env-overrides" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789461    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0514cc38-b832-48da-9ed3-24db150b634f" volumeName="kubernetes.io/projected/0514cc38-b832-48da-9ed3-24db150b634f-kube-api-access-mq26k" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789471    4466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97c08a18-b069-47a5-bdfd-7fca0c7767ae" volumeName="kubernetes.io/configmap/97c08a18-b069-47a5-bdfd-7fca0c7767ae-iptables-alerter-script" seLinuxMountContext=""
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789482    4466 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789490    4466 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789677    4466 factory.go:153] Registering CRI-O factory
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789691    4466 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789740    4466 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789749    4466 factory.go:55] Registering systemd factory
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789757    4466 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789772    4466 factory.go:103] Registering Raw factory
Oct 30 23:41:34.789781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.789784    4466 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:41:34.792896 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.790200    4466 manager.go:319] Starting recovery of all containers
Oct 30 23:41:34.818291 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.818265    4466 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:41:34.819462 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.819441    4466 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:41:34.819462 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.819463    4466 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:41:34.819656 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.819482    4466 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:41:34.819656 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.819526    4466 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:41:34.844104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.844066    4466 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="27ae4a377872128c3267642cc0b1ed689266a9536a354359834e20608f4e9e2e" exitCode=0
Oct 30 23:41:34.844104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.844101    4466 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="1bf5afd2ce79862f2617e3ce2bb6c6b3f20d13f44ef6ed3107f736b4ac5cdc0f" exitCode=0
Oct 30 23:41:34.844264 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.844114    4466 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="dd95a02a0ea800524f71fc61589360aa3a70a1c9f24b4fbc7d2daa943a97ebd8" exitCode=0
Oct 30 23:41:34.844264 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.844128    4466 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="a634150600d67af9479bb8c5d3068cc04ca6eee83463ebb610dfe6e2019050ad" exitCode=0
Oct 30 23:41:34.844264 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.844137    4466 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="4bd45d2ac8c8181db04e88634764e7dc7d2b25116377662012a0611b8fddb76d" exitCode=0
Oct 30 23:41:34.844264 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.844146    4466 generic.go:356] "Generic (PLEG): container finished" podID="44fe22ce-a102-4f4f-9f42-6c80dae9c9d6" containerID="a60124edde9bc20711720a321fe62ffa0191bbf18b9a3abbddfe2d655d2c3458" exitCode=0
Oct 30 23:41:34.857074 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.857054    4466 manager.go:324] Recovery completed
Oct 30 23:41:34.857429 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.857401    4466 generic.go:356] "Generic (PLEG): container finished" podID="6f30de5e28d2739412bf74e36d5e3bf9" containerID="c289690828346d4b9a49cf2e3e9ef145b61a8f85ee96af67c423e8fdca402f1f" exitCode=0
Oct 30 23:41:34.869955 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.869940    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.871180 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.871165    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.871266 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.871192    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.871266 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.871204    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.872476 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872451    4466 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:41:34.872476 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872463    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.872651 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872466    4466 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:41:34.872651 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872521    4466 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:41:34.872758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872706    4466 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 23:41:34.872758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872723    4466 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 23:41:34.872758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872744    4466 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 23:41:34.872758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872752    4466 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 23:41:34.872758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872759    4466 policy_none.go:50] "Start"
Oct 30 23:41:34.872975 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872784    4466 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:41:34.872975 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.872798    4466 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:41:34.873268 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.873253    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.873317 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.873282    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.873317 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.873292    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.886301 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.886251    4466 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 23:41:34.886301 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.886273    4466 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 23:41:34.886301 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.886281    4466 policy_none.go:44] "Start"
Oct 30 23:41:34.887272 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.887251    4466 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found"
Oct 30 23:41:34.890165 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.890142    4466 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:41:34.892740 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.892727    4466 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:41:34.892822 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.892754    4466 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:41:34.892822 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.892765    4466 server.go:87] "Starting device plugin registration server"
Oct 30 23:41:34.893005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.892992    4466 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:41:34.893058 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.893006    4466 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:41:34.893143 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.893128    4466 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:41:34.893749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.893200    4466 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:41:34.893749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.893233    4466 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:41:34.893749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.893237    4466 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:41:34.893749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.893277    4466 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:41:34.893224837Z","UUID":"f59f77d3-23db-4cbc-aded-0f8e3e481e19","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:41:34.894600 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.894581    4466 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:41:34.894725 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.894708    4466 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-123.us-west-2.compute.internal\" not found"
Oct 30 23:41:34.894725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.894661    4466 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:41:34.894868 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.894735    4466 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:41:34.920105 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.919650    4466 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"]
Oct 30 23:41:34.920190 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.920179    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.921991 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.921974    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.922055 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.922002    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.922055 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.922013    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.922236 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.922225    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.922363 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.922349    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.923204 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923185    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.923287 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923209    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.923287 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923232    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.923287 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923241    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.923287 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923215    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.923586 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923303    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.923586 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.923379    4466 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:34.923709 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923699    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.923764 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923694    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"6f30de5e28d2739412bf74e36d5e3bf9","Type":"ContainerStarted","Data":"37f2a9884952984032c416d7571957c409ea6850a0ab24639b8a7c0c9eef393f"}
Oct 30 23:41:34.923764 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923739    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"6f30de5e28d2739412bf74e36d5e3bf9","Type":"ContainerDied","Data":"c289690828346d4b9a49cf2e3e9ef145b61a8f85ee96af67c423e8fdca402f1f"}
Oct 30 23:41:34.923764 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923759    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"6f30de5e28d2739412bf74e36d5e3bf9","Type":"ContainerStarted","Data":"9706fcdb1b62aca0badca53f35749aae3280e833ffdaad3b2f170473bd95e553"}
Oct 30 23:41:34.923890 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923784    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"e32b577676385ef454846283b75daa6e","Type":"ContainerStarted","Data":"a2002ecafe3437b62ab8b9885222cbe81ceecaa7b49823e855b69d31848fb01f"}
Oct 30 23:41:34.923890 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.923803    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal" event={"ID":"e32b577676385ef454846283b75daa6e","Type":"ContainerStarted","Data":"85293fbffb2b5c8ec804f4b2db48fe64ab870c440d767f4a9478f6d1864e8a1f"}
Oct 30 23:41:34.924618 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.924599    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.924721 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.924664    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.924721 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.924674    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.924820 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:34.924810    4466 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-123.us-west-2.compute.internal\" not found" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:34.924875 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.924862    4466 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:34.967613 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.967590    4466 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:34.972106 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.972089    4466 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:34.972203 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.972158    4466 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 23:41:34.988320 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.988299    4466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:34.994466 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.994452    4466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:34.995513 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.995499    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:34.995559 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.995526    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:34.995559 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.995536    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:34.995621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:34.995566    4466 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.018964 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.018936    4466 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:35.060122 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.060100    4466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:41:35.060217 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:35.060159    4466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.060217 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.060179    4466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.072295 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.072268    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e32b577676385ef454846283b75daa6e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"e32b577676385ef454846283b75daa6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.072391 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.072308    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.072391 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.072337    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.109351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.109332    4466 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.109404 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.109385    4466 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.109875 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.109861    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeNotReady"
Oct 30 23:41:35.109942 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.109886    4466 setters.go:546] "Node became not ready" node="ip-10-0-131-123.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T23:41:35Z","lastTransitionTime":"2026-10-30T23:41:35Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 23:41:35.116724 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.116708    4466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:41:35.116776 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:35.116747    4466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.139561 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.139504    4466 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-123.us-west-2.compute.internal" is owned by different node (oldNodeID="37e1617c-e37f-4bd0-af4f-573e8069f2c0", newNodeID=""), cleaning up...
Oct 30 23:41:35.172651 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.172614    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e32b577676385ef454846283b75daa6e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"e32b577676385ef454846283b75daa6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.172742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.172668    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.172742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.172695    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.172742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.172708    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e32b577676385ef454846283b75daa6e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"e32b577676385ef454846283b75daa6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.172848 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.172743    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.172848 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.172762    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f30de5e28d2739412bf74e36d5e3bf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" (UID: \"6f30de5e28d2739412bf74e36d5e3bf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.375418 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.375394    4466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-123.us-west-2.compute.internal" event="NodeReady"
Oct 30 23:41:35.375536 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.375469    4466 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 23:41:35.770000 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.769979    4466 apiserver.go:52] "Watching apiserver"
Oct 30 23:41:35.863604 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.863581    4466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.863733 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.863726    4466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.870537 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.870514    4466 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:41:35.871747 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.871719    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-d7f6p","openshift-insights/insights-runtime-extractor-cm5bp","openshift-cluster-node-tuning-operator/tuned-9npkw","openshift-multus/multus-additional-cni-plugins-j7mw9","kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p","openshift-dns/dns-default-hx47s","openshift-ingress-canary/ingress-canary-45bcz","openshift-network-diagnostics/network-check-target-t5xw7","openshift-service-ca/service-ca-5fffc97bdf-2k2kl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-7xjgb","openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f","openshift-ovn-kubernetes/ovnkube-node-mdhf9","openshift-dns/node-resolver-kmhdt","openshift-image-registry/node-ca-ld74g","openshift-multus/network-metrics-daemon-hrmp7","kube-system/global-pull-secret-syncer-gl7bk","openshift-multus/multus-h2w25"]
Oct 30 23:41:35.874174 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.874148    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:35.876348 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.876330    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.876795 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.876779    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.888782 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.888764    4466 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:41:35.919618 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.919599    4466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:41:35.919684 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:35.919652    4466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.920454 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.920441    4466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:41:35.920487 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:35.920475    4466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-123.us-west-2.compute.internal"
Oct 30 23:41:35.924009 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.923990    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.924092 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924032    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.924092 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924035    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.924092 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924055    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.924092 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924063    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.924276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924117    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.924276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924133    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:41:35.924276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924148    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.924276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924172    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.924276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924184    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.924276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924218    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:41:35.924493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924335    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.924493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.924460    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:41:35.925070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925053    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\""
Oct 30 23:41:35.925240 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925225    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\""
Oct 30 23:41:35.925305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925258    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72946\""
Oct 30 23:41:35.925305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925288    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xnb7d\""
Oct 30 23:41:35.925408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925378    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwz57\""
Oct 30 23:41:35.925408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925406    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:41:35.925408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925411    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:41:35.925538 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925440    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7rnf\""
Oct 30 23:41:35.925820 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925808    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\""
Oct 30 23:41:35.925958 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.925814    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:41:35.961432 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.961412    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w5vpr\""
Oct 30 23:41:35.962013 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.961998    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.962866 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.962843    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:41:35.963153 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963139    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.963210 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963194    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:41:35.963305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963294    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.963340 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963325    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.963418 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963406    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.963721 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963708    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:41:35.963969 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963955    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.964015 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.963970    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:41:35.964074 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.964058    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:41:35.964129 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.964068    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.964129 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.964062    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:41:35.964217 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.964192    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-28w6g\""
Oct 30 23:41:35.964264 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.964230    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.965117 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.965106    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:41:35.966338 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.966323    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-znsw7\""
Oct 30 23:41:35.966435 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.966341    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6s2q2\""
Oct 30 23:41:35.966435 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.966343    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hcpbw\""
Oct 30 23:41:35.966435 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.966379    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnlbs\""
Oct 30 23:41:35.968853 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.968838    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:41:35.969030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.969016    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:41:35.969137 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.969123    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ksfp2\""
Oct 30 23:41:35.969137 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.969134    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvkr7\""
Oct 30 23:41:35.970419 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970407    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:41:35.970570 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970557    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:41:35.970653 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970624    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:41:35.970699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970651    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hnph4\""
Oct 30 23:41:35.970729 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970724    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.970865 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970854    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.970933 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970915    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:41:35.970987 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970918    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.970987 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970945    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zg485\""
Oct 30 23:41:35.970987 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970952    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.970987 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970963    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:41:35.970987 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.970969    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:41:35.971211 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.971037    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:41:35.971211 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.971103    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:41:35.971211 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.971149    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:41:35.976404 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976386    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-bin\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.976484 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976412    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-cni-binary-copy\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.976484 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976429    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-socket-dir-parent\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.976484 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976459    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvvz\" (UniqueName: \"kubernetes.io/projected/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-kube-api-access-fjvvz\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:35.976627 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976519    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-kubelet\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.976627 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976565    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.976738 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976677    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.976738 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976712    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97c08a18-b069-47a5-bdfd-7fca0c7767ae-host-slash\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:35.976849 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976755    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq26k\" (UniqueName: \"kubernetes.io/projected/0514cc38-b832-48da-9ed3-24db150b634f-kube-api-access-mq26k\") pod \"migrator-66df5d496b-vt56f\" (UID: \"0514cc38-b832-48da-9ed3-24db150b634f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:35.976849 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976783    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-system-cni-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.976849 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976817    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-cni-binary-copy\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.976849 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976836    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.977020 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976869    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.977020 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.976943    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2j2\" (UniqueName: \"kubernetes.io/projected/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-kube-api-access-ff2j2\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:35.977313 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977292    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.977465 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977301    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.977557 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977292    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.977660 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977567    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.977771 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977756    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d645c1da-d44f-4765-bcdf-afe19821b63c-serviceca\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:35.977874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977862    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-run\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.977966 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.977952    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/685220ca-ec2c-4955-91eb-53a4d5a9c62b-hosts-file\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:35.978101 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978087    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-tmp\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.978203 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978190    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-etc-kubernetes\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.978305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978291    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-kubelet-config\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:35.978393 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978382    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-agent-certs\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:35.978477 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978466    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-sys\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.978559 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978548    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-lib-modules\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.978658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978645    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-tuned\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.978750 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978738    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-multus\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.978848 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978836    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:35.978932 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.978921    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.979017 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979006    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsst\" (UniqueName: \"kubernetes.io/projected/685220ca-ec2c-4955-91eb-53a4d5a9c62b-kube-api-access-pqsst\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:35.979102 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979091    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-modprobe-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.979192 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979181    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-netd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.979276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979264    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cnibin\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.979364 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979352    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysconfig\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.979458 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979445    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-env-overrides\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.979537 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979526    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.979617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979606    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-multus-certs\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.979720 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979707    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-konnectivity-ca\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:35.979822 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979810    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97c08a18-b069-47a5-bdfd-7fca0c7767ae-iptables-alerter-script\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:35.979975 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979959    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-tuned\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.980058 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.979964    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f8297f-f076-43cb-ba9f-93a455101c9f-tmp\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.980211 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980171    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-env-overrides\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.980342 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980229    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-konnectivity-ca\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:35.980342 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980281    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e895adf2-43c1-46f2-b2de-a3bddfd08f0c-agent-certs\") pod \"konnectivity-agent-d7f6p\" (UID: \"e895adf2-43c1-46f2-b2de-a3bddfd08f0c\") " pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:35.980522 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980043    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-host\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.980908 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980887    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:35.981000 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980926    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97c08a18-b069-47a5-bdfd-7fca0c7767ae-iptables-alerter-script\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:35.981000 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980938    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deaff92c-3bfd-4b69-a59d-3bf4d6716624-tmp-dir\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.981000 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.980975    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.981135 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981003    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-log-socket\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.981135 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981033    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-script-lib\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.981135 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981037    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deaff92c-3bfd-4b69-a59d-3bf4d6716624-tmp-dir\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.981135 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981072    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bg7r\" (UniqueName: \"kubernetes.io/projected/24273fe9-3b2a-421c-bec7-26c6af951dcb-kube-api-access-7bg7r\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.981308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981137    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ba94fb-766a-414b-a027-b378c42881e3-data-volume\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.981308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981166    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9h7\" (UniqueName: \"kubernetes.io/projected/a7f8297f-f076-43cb-ba9f-93a455101c9f-kube-api-access-wd9h7\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.981308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981192    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-netns\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.981308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981223    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d645c1da-d44f-4765-bcdf-afe19821b63c-serviceca\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:35.981308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981247    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-script-lib\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.981308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981254    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ba94fb-766a-414b-a027-b378c42881e3-data-volume\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981323    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-key\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981360    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981481    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-key\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981488    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981519    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981540    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-device-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981581    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981606    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-cnibin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.981665 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981654    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrfr\" (UniqueName: \"kubernetes.io/projected/deaff92c-3bfd-4b69-a59d-3bf4d6716624-kube-api-access-8zrfr\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981681    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685220ca-ec2c-4955-91eb-53a4d5a9c62b-tmp-dir\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981703    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-conf\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981727    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-slash\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981751    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95ec5cc3-2b66-46c0-9732-228496f671f4-cert\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981773    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-config\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981774    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685220ca-ec2c-4955-91eb-53a4d5a9c62b-tmp-dir\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981804    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-kubernetes\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981835    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-systemd-units\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981894    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-system-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981923    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-bin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981943    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovnkube-config\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.981962    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/97c08a18-b069-47a5-bdfd-7fca0c7767ae-kube-api-access-tlm89\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982000    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95ec5cc3-2b66-46c0-9732-228496f671f4-cert\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982003    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.982030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982027    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-systemd\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982048    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982100    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-etc-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982118    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qrj\" (UniqueName: \"kubernetes.io/projected/f6b0becd-1e63-4dc2-a01e-3415794c4b46-kube-api-access-68qrj\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982134    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-kubelet\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982148    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-dbus\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982163    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grks\" (UniqueName: \"kubernetes.io/projected/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kube-api-access-2grks\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982209    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-var-lib-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982238    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovn-node-metrics-cert\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982266    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-k8s-cni-cncf-io\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982321    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-hostroot\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982346    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6b0becd-1e63-4dc2-a01e-3415794c4b46-ovn-node-metrics-cert\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982356    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvd6\" (UniqueName: \"kubernetes.io/projected/69ba94fb-766a-414b-a027-b378c42881e3-kube-api-access-jdvd6\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982379    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d645c1da-d44f-4765-bcdf-afe19821b63c-host\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982394    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-systemd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982412    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-daemon-config\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982452    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982470    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982484    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-netns\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982499    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-ovn\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.982611 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982531    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-conf-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982663    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdcp\" (UniqueName: \"kubernetes.io/projected/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-kube-api-access-jjdcp\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982695    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45dfef13-d5e6-4bff-98d2-47032ef6f624-original-pull-secret\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982710    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ba94fb-766a-414b-a027-b378c42881e3-crio-socket\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982738    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982739    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-daemon-config\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982780    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982808    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-var-lib-kubelet\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982836    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deaff92c-3bfd-4b69-a59d-3bf4d6716624-config-volume\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982866    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfhn\" (UniqueName: \"kubernetes.io/projected/95ec5cc3-2b66-46c0-9732-228496f671f4-kube-api-access-4sfhn\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982897    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ba94fb-766a-414b-a027-b378c42881e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982918    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-node-log\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982935    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.982966    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqds5\" (UniqueName: \"kubernetes.io/projected/d645c1da-d44f-4765-bcdf-afe19821b63c-kube-api-access-qqds5\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.983010    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/deaff92c-3bfd-4b69-a59d-3bf4d6716624-config-volume\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:35.983408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:35.983155    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.034613 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.034567    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvvz\" (UniqueName: \"kubernetes.io/projected/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-kube-api-access-fjvvz\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:36.036183 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.036168    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2j2\" (UniqueName: \"kubernetes.io/projected/d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5-kube-api-access-ff2j2\") pod \"service-ca-5fffc97bdf-2k2kl\" (UID: \"d7bf7af6-4d8c-4b3a-960f-56fbb356e7c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2k2kl"
Oct 30 23:41:36.037059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.037044    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsst\" (UniqueName: \"kubernetes.io/projected/685220ca-ec2c-4955-91eb-53a4d5a9c62b-kube-api-access-pqsst\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:36.037133 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.037119    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxnf\" (UniqueName: \"kubernetes.io/projected/3b67fb8d-7158-4441-9f05-5bd7aebdcf5c-kube-api-access-zgxnf\") pod \"network-check-target-t5xw7\" (UID: \"3b67fb8d-7158-4441-9f05-5bd7aebdcf5c\") " pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:36.038100 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.038085    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9h7\" (UniqueName: \"kubernetes.io/projected/a7f8297f-f076-43cb-ba9f-93a455101c9f-kube-api-access-wd9h7\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.038497 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.038475    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grks\" (UniqueName: \"kubernetes.io/projected/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kube-api-access-2grks\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.041503 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.041478    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bg7r\" (UniqueName: \"kubernetes.io/projected/24273fe9-3b2a-421c-bec7-26c6af951dcb-kube-api-access-7bg7r\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.042014 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.041999    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qrj\" (UniqueName: \"kubernetes.io/projected/f6b0becd-1e63-4dc2-a01e-3415794c4b46-kube-api-access-68qrj\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.042144 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.042128    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfhn\" (UniqueName: \"kubernetes.io/projected/95ec5cc3-2b66-46c0-9732-228496f671f4-kube-api-access-4sfhn\") pod \"ingress-canary-45bcz\" (UID: \"95ec5cc3-2b66-46c0-9732-228496f671f4\") " pod="openshift-ingress-canary/ingress-canary-45bcz"
Oct 30 23:41:36.043437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.043418    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvd6\" (UniqueName: \"kubernetes.io/projected/69ba94fb-766a-414b-a027-b378c42881e3-kube-api-access-jdvd6\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:36.043496 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.043473    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdcp\" (UniqueName: \"kubernetes.io/projected/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-kube-api-access-jjdcp\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.043496 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.043485    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq26k\" (UniqueName: \"kubernetes.io/projected/0514cc38-b832-48da-9ed3-24db150b634f-kube-api-access-mq26k\") pod \"migrator-66df5d496b-vt56f\" (UID: \"0514cc38-b832-48da-9ed3-24db150b634f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vt56f"
Oct 30 23:41:36.043551 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.043497    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrfr\" (UniqueName: \"kubernetes.io/projected/deaff92c-3bfd-4b69-a59d-3bf4d6716624-kube-api-access-8zrfr\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:36.043685 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.043671    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm89\" (UniqueName: \"kubernetes.io/projected/97c08a18-b069-47a5-bdfd-7fca0c7767ae-kube-api-access-tlm89\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:36.044882 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.044868    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqds5\" (UniqueName: \"kubernetes.io/projected/d645c1da-d44f-4765-bcdf-afe19821b63c-kube-api-access-qqds5\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:36.083373 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083353    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-systemd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083416 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083381    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.083416 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083395    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-netns\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083412    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-ovn\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083435    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-conf-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.083493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083464    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-systemd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083475    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-conf-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.083493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083466    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-ovn\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083494    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083506    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-netns\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083525    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ba94fb-766a-414b-a027-b378c42881e3-crio-socket\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083552    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083573    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-var-lib-kubelet\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083580    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ba94fb-766a-414b-a027-b378c42881e3-crio-socket\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083588    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-node-log\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083603    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-bin\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083618    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-socket-dir-parent\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083648    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-kubelet\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083657    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-var-lib-kubelet\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083670    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.083680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083675    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-bin\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083678    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-kubelet\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083694    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083695    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-socket-dir-parent\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083694    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97c08a18-b069-47a5-bdfd-7fca0c7767ae-host-slash\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083649    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-node-log\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083657    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083728    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97c08a18-b069-47a5-bdfd-7fca0c7767ae-host-slash\") pod \"iptables-alerter-7xjgb\" (UID: \"97c08a18-b069-47a5-bdfd-7fca0c7767ae\") " pod="openshift-network-operator/iptables-alerter-7xjgb"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083764    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-system-cni-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083791    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083819    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083828    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-system-cni-dir\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083843    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083866    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083866    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-run\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.083873    4466 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083890    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/685220ca-ec2c-4955-91eb-53a4d5a9c62b-hosts-file\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083905    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-run\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083915    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-etc-kubernetes\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083930    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-kubelet-config\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.083947    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:36.583932449 +0000 UTC m=+1.874480763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083947    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/685220ca-ec2c-4955-91eb-53a4d5a9c62b-hosts-file\") pod \"node-resolver-kmhdt\" (UID: \"685220ca-ec2c-4955-91eb-53a4d5a9c62b\") " pod="openshift-dns/node-resolver-kmhdt"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083954    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-kubelet-config\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083944    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083973    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-etc-kubernetes\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.083998    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-sys\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084012    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-sys\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084014    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-lib-modules\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084030    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-multus\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084051    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084068    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-modprobe-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084082    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-netd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084113    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-cni-netd\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084118    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084141    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-multus\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084152    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-lib-modules\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.084141 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084166    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cnibin\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084145    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44fe22ce-a102-4f4f-9f42-6c80dae9c9d6-cnibin\") pod \"multus-additional-cni-plugins-j7mw9\" (UID: \"44fe22ce-a102-4f4f-9f42-6c80dae9c9d6\") " pod="openshift-multus/multus-additional-cni-plugins-j7mw9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084188    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-modprobe-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084191    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysconfig\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084226    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysconfig\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084232    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084257    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-multus-certs\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084265    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-multus-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084281    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-host\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084303    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084326    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084346    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-host\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084348    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-log-socket\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084364    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-multus-certs\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084378    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-log-socket\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084384    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-run-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084388    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-netns\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084411    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084425    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-device-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084439    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084463    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-cnibin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084475    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-device-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084479    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-netns\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084490    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-conf\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084499    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgn5p\" (UID: \"555bcb1e-9ce9-46e1-ab6b-c4cb08127e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgn5p"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084507    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-cnibin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084509    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-slash\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084528    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-kubernetes\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084542    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-systemd-units\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084539    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084563    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-system-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084579    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-systemd-units\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084580    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-kubernetes\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084602    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-conf\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084608    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-bin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084604    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-system-cni-dir\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084613    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-host-slash\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084653    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084589    4466 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084676    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-cni-bin\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084674    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-systemd\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084707    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084720    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-etc-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084720    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-systemd\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084758    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f8297f-f076-43cb-ba9f-93a455101c9f-etc-sysctl-d\") pod \"tuned-9npkw\" (UID: \"a7f8297f-f076-43cb-ba9f-93a455101c9f\") " pod="openshift-cluster-node-tuning-operator/tuned-9npkw"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.084772    4466 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084790    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-kubelet\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.084806    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:36.584792688 +0000 UTC m=+1.875340984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084773    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-etc-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084824    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-var-lib-kubelet\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084839    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-dbus\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084865    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-var-lib-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084880    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-k8s-cni-cncf-io\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084895    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45dfef13-d5e6-4bff-98d2-47032ef6f624-dbus\") pod \"global-pull-secret-syncer-gl7bk\" (UID: \"45dfef13-d5e6-4bff-98d2-47032ef6f624\") " pod="kube-system/global-pull-secret-syncer-gl7bk"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084894    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-hostroot\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084941    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6b0becd-1e63-4dc2-a01e-3415794c4b46-var-lib-openvswitch\") pod \"ovnkube-node-mdhf9\" (UID: \"f6b0becd-1e63-4dc2-a01e-3415794c4b46\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084946    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-host-run-k8s-cni-cncf-io\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084959    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d645c1da-d44f-4765-bcdf-afe19821b63c-host\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.084987    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24273fe9-3b2a-421c-bec7-26c6af951dcb-hostroot\") pod \"multus-h2w25\" (UID: \"24273fe9-3b2a-421c-bec7-26c6af951dcb\") " pod="openshift-multus/multus-h2w25"
Oct 30 23:41:36.085398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.085016    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d645c1da-d44f-4765-bcdf-afe19821b63c-host\") pod \"node-ca-ld74g\" (UID: \"d645c1da-d44f-4765-bcdf-afe19821b63c\") " pod="openshift-image-registry/node-ca-ld74g"
Oct 30 23:41:36.087353 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.086691    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98-metrics-certs\") pod \"network-metrics-daemon-hrmp7\" (UID: \"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98\") " pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:36.174432 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.174408    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrmp7"
Oct 30 23:41:36.532455 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.532424    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hrmp7"]
Oct 30 23:41:36.536121 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:36.536090    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe518c1_bb3e_4bd8_89e5_b5b1f7be4c98.slice/crio-23f1de98bbac26407012c08cfc64e877fbb45c951132e954fb849a961a28b6f7 WatchSource:0}: Error finding container 23f1de98bbac26407012c08cfc64e877fbb45c951132e954fb849a961a28b6f7: Status 404 returned error can't find the container with id 23f1de98bbac26407012c08cfc64e877fbb45c951132e954fb849a961a28b6f7
Oct 30 23:41:36.538321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.538301    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:41:36.588504 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.588476    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:36.588621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.588526    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:36.588621 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.588608    4466 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:36.588742 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.588678    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:37.588664204 +0000 UTC m=+2.879212516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:36.588742 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.588615    4466 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:36.588833 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:36.588752    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:37.588736916 +0000 UTC m=+2.879285217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:36.866423 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:36.866392    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrmp7" event={"ID":"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98","Type":"ContainerStarted","Data":"23f1de98bbac26407012c08cfc64e877fbb45c951132e954fb849a961a28b6f7"}
Oct 30 23:41:37.594731 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:37.594700    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:37.594878 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:37.594753    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:37.594878 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:37.594845    4466 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:41:37.594958 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:37.594896    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls podName:deaff92c-3bfd-4b69-a59d-3bf4d6716624 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:39.594885379 +0000 UTC m=+4.885433689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls") pod "dns-default-hx47s" (UID: "deaff92c-3bfd-4b69-a59d-3bf4d6716624") : secret "dns-default-metrics-tls" not found
Oct 30 23:41:37.594958 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:37.594845    4466 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:37.595047 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:37.594999    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:39.594980766 +0000 UTC m=+4.885529068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:38.874032 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:38.873996    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrmp7" event={"ID":"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98","Type":"ContainerStarted","Data":"27e75f6f54d3c15a9d75ede9aba15f1bd6d5f01444048070ebd4bf8bf87b4a42"}
Oct 30 23:41:38.874577 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:38.874045    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrmp7" event={"ID":"4fe518c1-bb3e-4bd8-89e5-b5b1f7be4c98","Type":"ContainerStarted","Data":"eeda6b525f3376ed447a4bf1324642bd80360a04da9f90a58b38d7cde837c5a2"}
Oct 30 23:41:39.280817 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:39.280750    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:41:39.606831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:39.606803    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:39.606998 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:39.606854    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:39.606998 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:39.606967    4466 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:39.607113 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:41:39.607029    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls podName:69ba94fb-766a-414b-a027-b378c42881e3 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:43.607012003 +0000 UTC m=+8.897560313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cm5bp" (UID: "69ba94fb-766a-414b-a027-b378c42881e3") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:41:39.609009 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:39.608988    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/deaff92c-3bfd-4b69-a59d-3bf4d6716624-metrics-tls\") pod \"dns-default-hx47s\" (UID: \"deaff92c-3bfd-4b69-a59d-3bf4d6716624\") " pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:39.777829 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:39.777800    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:40.138114 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:40.138018    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hx47s"]
Oct 30 23:41:40.141725 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:40.141696    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaff92c_3bfd_4b69_a59d_3bf4d6716624.slice/crio-bbd2f54b256d8b0b41c6b39a95e09eefa2bc9b7619ff88e6b40a9cab160a9128 WatchSource:0}: Error finding container bbd2f54b256d8b0b41c6b39a95e09eefa2bc9b7619ff88e6b40a9cab160a9128: Status 404 returned error can't find the container with id bbd2f54b256d8b0b41c6b39a95e09eefa2bc9b7619ff88e6b40a9cab160a9128
Oct 30 23:41:40.881227 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:40.881193    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hx47s" event={"ID":"deaff92c-3bfd-4b69-a59d-3bf4d6716624","Type":"ContainerStarted","Data":"bbd2f54b256d8b0b41c6b39a95e09eefa2bc9b7619ff88e6b40a9cab160a9128"}
Oct 30 23:41:41.284468 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:41.283131    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:42.890418 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:42.890389    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hx47s" event={"ID":"deaff92c-3bfd-4b69-a59d-3bf4d6716624","Type":"ContainerStarted","Data":"854409cda18b796c9b700d853954613b9c39a244bc2214466e2b912a039178fe"}
Oct 30 23:41:42.890795 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:42.890437    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hx47s" event={"ID":"deaff92c-3bfd-4b69-a59d-3bf4d6716624","Type":"ContainerStarted","Data":"b0e309cdc40e2da7f6f81f9c556adfe491bc981f479df4f44e26a8ca97324583"}
Oct 30 23:41:42.890795 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:42.890500    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:43.361337 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:43.361304    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:41:43.361498 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:43.361490    4466 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:41:43.633199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:43.633132    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:43.635243 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:43.635221    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ba94fb-766a-414b-a027-b378c42881e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cm5bp\" (UID: \"69ba94fb-766a-414b-a027-b378c42881e3\") " pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:43.677153 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:43.677127    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cm5bp"
Oct 30 23:41:44.037504 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:44.037437    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cm5bp"]
Oct 30 23:41:44.040547 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:44.040520    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ba94fb_766a_414b_a027_b378c42881e3.slice/crio-05711bc2d86b6bc4a996c44886a5a71d5622fb0a3b543a58634b477b817949d8 WatchSource:0}: Error finding container 05711bc2d86b6bc4a996c44886a5a71d5622fb0a3b543a58634b477b817949d8: Status 404 returned error can't find the container with id 05711bc2d86b6bc4a996c44886a5a71d5622fb0a3b543a58634b477b817949d8
Oct 30 23:41:44.900583 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:44.900549    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cm5bp" event={"ID":"69ba94fb-766a-414b-a027-b378c42881e3","Type":"ContainerStarted","Data":"e4ab1ecae36b41966b3db87cdabed5097741e24108e0f85900e2e91737c9d63f"}
Oct 30 23:41:44.900769 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:44.900606    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cm5bp" event={"ID":"69ba94fb-766a-414b-a027-b378c42881e3","Type":"ContainerStarted","Data":"05711bc2d86b6bc4a996c44886a5a71d5622fb0a3b543a58634b477b817949d8"}
Oct 30 23:41:45.022871 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.022851    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7f6p"
Oct 30 23:41:45.173990 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.173923    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-568b89444d-55qld"]
Oct 30 23:41:45.176555 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.176540    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.244991 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.244960    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-registry-tls\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245024    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-trusted-ca\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245044    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-ca-trust-extracted\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245062    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwr5\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-kube-api-access-jgwr5\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245100    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-image-registry-private-configuration\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245118    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-bound-sa-token\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245135    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-registry-certificates\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.245154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.245155    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-installation-pull-secrets\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.323321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.321975    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 23:41:45.345477 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345448    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-image-registry-private-configuration\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345484    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-bound-sa-token\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345505    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-registry-certificates\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345524    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-installation-pull-secrets\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345553    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-registry-tls\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345585    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-trusted-ca\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345605    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-ca-trust-extracted\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.345899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.345628    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwr5\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-kube-api-access-jgwr5\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.346047 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.346023    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-ca-trust-extracted\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.346319 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.346295    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-registry-certificates\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.346531 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.346510    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-trusted-ca\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.364929 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.364878    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 23:41:45.365131 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.365108    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 23:41:45.365239 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.365157    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pkxn2\""
Oct 30 23:41:45.365288 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.365243    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 23:41:45.366403 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.366380    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-568b89444d-55qld"]
Oct 30 23:41:45.368135 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.368113    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-image-registry-private-configuration\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.368238 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.368219    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-registry-tls\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.368603 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.368584    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-installation-pull-secrets\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.405207 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.405173    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwr5\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-kube-api-access-jgwr5\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.405343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.405322    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a-bound-sa-token\") pod \"image-registry-568b89444d-55qld\" (UID: \"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a\") " pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.487737 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.487716    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:45.905165 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.905137    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cm5bp" event={"ID":"69ba94fb-766a-414b-a027-b378c42881e3","Type":"ContainerStarted","Data":"09be44a604b1c248e097ffa517f21bd51e3c5ec3abef8b653fd43cbc49965213"}
Oct 30 23:41:45.940949 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:45.940925    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-568b89444d-55qld"]
Oct 30 23:41:45.944172 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:45.944138    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4a1ad8_ddf9_4dff_a21c_38fd6fcc0b7a.slice/crio-1a17b0955839c6dd2bcd3ea39f188ed5ca3bedd071f980da8665a13df94b80b2 WatchSource:0}: Error finding container 1a17b0955839c6dd2bcd3ea39f188ed5ca3bedd071f980da8665a13df94b80b2: Status 404 returned error can't find the container with id 1a17b0955839c6dd2bcd3ea39f188ed5ca3bedd071f980da8665a13df94b80b2
Oct 30 23:41:46.909825 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:46.909792    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-568b89444d-55qld" event={"ID":"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a","Type":"ContainerStarted","Data":"f232bfacbaf437477caef1249c76aede2726236657503e0842d275f613ef4b17"}
Oct 30 23:41:46.910286 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:46.909841    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-568b89444d-55qld" event={"ID":"1c4a1ad8-ddf9-4dff-a21c-38fd6fcc0b7a","Type":"ContainerStarted","Data":"1a17b0955839c6dd2bcd3ea39f188ed5ca3bedd071f980da8665a13df94b80b2"}
Oct 30 23:41:46.910286 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:46.909923    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:41:47.023759 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:47.023307    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-568b89444d-55qld" podStartSLOduration=2.023289519 podStartE2EDuration="2.023289519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:47.022903827 +0000 UTC m=+12.313452147" watchObservedRunningTime="2026-10-30 23:41:47.023289519 +0000 UTC m=+12.313837835"
Oct 30 23:41:47.916109 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:47.916036    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cm5bp" event={"ID":"69ba94fb-766a-414b-a027-b378c42881e3","Type":"ContainerStarted","Data":"8cfe7b83eff012f5db85fac01db2254808db3a550554b74ada0f8bbeaab277b3"}
Oct 30 23:41:49.703106 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.703065    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"]
Oct 30 23:41:49.706123 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.706108    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:49.773692 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.773665    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84833d2f-e775-46d1-be8c-a56a1c9c0b0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t8wch\" (UID: \"84833d2f-e775-46d1-be8c-a56a1c9c0b0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:49.801686 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.801663    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 23:41:49.801803 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.801690    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-72sq6\""
Oct 30 23:41:49.802285 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.802267    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"]
Oct 30 23:41:49.874742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.874717    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84833d2f-e775-46d1-be8c-a56a1c9c0b0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t8wch\" (UID: \"84833d2f-e775-46d1-be8c-a56a1c9c0b0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:49.877001 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:49.876985    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84833d2f-e775-46d1-be8c-a56a1c9c0b0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t8wch\" (UID: \"84833d2f-e775-46d1-be8c-a56a1c9c0b0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:50.017471 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:50.017445    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:50.378300 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:50.378271    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"]
Oct 30 23:41:50.382151 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:41:50.382123    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84833d2f_e775_46d1_be8c_a56a1c9c0b0a.slice/crio-ae0c60e72ec411558bf24936cbf38d77393a84c1b49de587584fcba08d63e3db WatchSource:0}: Error finding container ae0c60e72ec411558bf24936cbf38d77393a84c1b49de587584fcba08d63e3db: Status 404 returned error can't find the container with id ae0c60e72ec411558bf24936cbf38d77393a84c1b49de587584fcba08d63e3db
Oct 30 23:41:50.925315 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:50.925288    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch" event={"ID":"84833d2f-e775-46d1-be8c-a56a1c9c0b0a","Type":"ContainerStarted","Data":"ae0c60e72ec411558bf24936cbf38d77393a84c1b49de587584fcba08d63e3db"}
Oct 30 23:41:52.895559 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:52.895526    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hx47s"
Oct 30 23:41:52.932866 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:52.932835    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch" event={"ID":"84833d2f-e775-46d1-be8c-a56a1c9c0b0a","Type":"ContainerStarted","Data":"eb4fb5c16db61edcecbd4d1a0217ece4e497492f098d3871defa47fa5f38a213"}
Oct 30 23:41:52.932992 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:52.932959    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:52.937208 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:52.937185    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch"
Oct 30 23:41:53.115435 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:41:53.115377    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t8wch" podStartSLOduration=2.434015875 podStartE2EDuration="4.115361893s" totalImagesPullingTime="1.681346018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:53.114673226 +0000 UTC m=+18.405221545" watchObservedRunningTime="2026-10-30 23:41:53.115361893 +0000 UTC m=+18.405910212"
Oct 30 23:42:00.081444 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:00.081414    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdhf9"
Oct 30 23:42:00.966678 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:00.966629    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slvxx"]
Oct 30 23:42:00.971053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:00.971036    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.020706 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.020674    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:42:01.044143 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044119    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-sys\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044247 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044146    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-metrics-client-ca\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044247 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044172    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044311 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044241    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-wtmp\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044311 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044277    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-tls\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044374 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044321    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-accelerators-collector-config\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044374 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044346    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-textfile\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044432 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044392    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqdr\" (UniqueName: \"kubernetes.io/projected/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-kube-api-access-5mqdr\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.044462 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.044436    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-root\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.067318 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.067290    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:42:01.067494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.067381    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 23:42:01.067494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.067458    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 23:42:01.067494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.067470    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 23:42:01.067494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.067469    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4224x\""
Oct 30 23:42:01.067494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.067498    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 23:42:01.145501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145474    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-accelerators-collector-config\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.145501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145504    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-textfile\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145526    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqdr\" (UniqueName: \"kubernetes.io/projected/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-kube-api-access-5mqdr\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145547    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-root\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145586    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-sys\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145608    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-metrics-client-ca\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145667    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145697    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-wtmp\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145709    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-root\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145729    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-tls\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145734    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-sys\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145897    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-wtmp\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146005 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.145914    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-textfile\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146397 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.146257    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-metrics-client-ca\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.146397 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.146285    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-accelerators-collector-config\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.147853 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.147830    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.147951 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.147934    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-node-exporter-tls\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.205877 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.205850    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqdr\" (UniqueName: \"kubernetes.io/projected/d48dc9d1-11d0-4633-9cf7-50f4229fdc2e-kube-api-access-5mqdr\") pod \"node-exporter-slvxx\" (UID: \"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e\") " pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.285675 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.285604    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slvxx"
Oct 30 23:42:01.296302 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:42:01.296278    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48dc9d1_11d0_4633_9cf7_50f4229fdc2e.slice/crio-7d1b960b8af28372da7d47ecc023120b7b980d22b85b9656b87647ffc339c9ee WatchSource:0}: Error finding container 7d1b960b8af28372da7d47ecc023120b7b980d22b85b9656b87647ffc339c9ee: Status 404 returned error can't find the container with id 7d1b960b8af28372da7d47ecc023120b7b980d22b85b9656b87647ffc339c9ee
Oct 30 23:42:01.961174 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:01.961138    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slvxx" event={"ID":"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e","Type":"ContainerStarted","Data":"7d1b960b8af28372da7d47ecc023120b7b980d22b85b9656b87647ffc339c9ee"}
Oct 30 23:42:03.968194 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:03.968157    4466 generic.go:356] "Generic (PLEG): container finished" podID="d48dc9d1-11d0-4633-9cf7-50f4229fdc2e" containerID="e55866bf4b762545df728f38e01a260985267137d85feb94b6908cc336200ffb" exitCode=0
Oct 30 23:42:03.968586 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:03.968208    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slvxx" event={"ID":"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e","Type":"ContainerDied","Data":"e55866bf4b762545df728f38e01a260985267137d85feb94b6908cc336200ffb"}
Oct 30 23:42:04.973835 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:04.973797    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slvxx" event={"ID":"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e","Type":"ContainerStarted","Data":"d7b79c1817d73ff13214664f0cbd13c284a643c27a835daceb8d4ab829296cd1"}
Oct 30 23:42:04.974328 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:04.973860    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slvxx" event={"ID":"d48dc9d1-11d0-4633-9cf7-50f4229fdc2e","Type":"ContainerStarted","Data":"002df11bdf63527f4b2ed1fe925034800d6a474d130680575d7ac0e920cb2039"}
Oct 30 23:42:05.089352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.089295    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slvxx" podStartSLOduration=3.352609292 podStartE2EDuration="5.089276961s" totalImagesPullingTime="1.673797471s" totalInitContainerRuntime="62.870198ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:05.087131954 +0000 UTC m=+30.377680271" watchObservedRunningTime="2026-10-30 23:42:05.089276961 +0000 UTC m=+30.379825283"
Oct 30 23:42:05.656845 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.656809    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kff85"]
Oct 30 23:42:05.682745 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.682713    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.711061 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.711038    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kff85"]
Oct 30 23:42:05.731833 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.731807    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 23:42:05.731936 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.731853    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-srvzs\""
Oct 30 23:42:05.779058 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.779032    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftkbl\" (UniqueName: \"kubernetes.io/projected/eda1ddc3-ab20-471e-b273-9fa66556e35f-kube-api-access-ftkbl\") pod \"monitoring-plugin-77f4f76594-kff85\" (UID: \"eda1ddc3-ab20-471e-b273-9fa66556e35f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.779187 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.779079    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eda1ddc3-ab20-471e-b273-9fa66556e35f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kff85\" (UID: \"eda1ddc3-ab20-471e-b273-9fa66556e35f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.795917 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.795891    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d948c677c-pt8sl"]
Oct 30 23:42:05.813050 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.813031    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.855153 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.855123    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d948c677c-pt8sl"]
Oct 30 23:42:05.862592 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.862559    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 23:42:05.862592 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.862573    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 23:42:05.862592 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.862573    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 23:42:05.862765 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.862598    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zt2b4\""
Oct 30 23:42:05.862765 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.862753    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 23:42:05.862821 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.862809    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5i1uej1i9nmh3\""
Oct 30 23:42:05.880221 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880198    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-audit-log\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880303 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880238    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftkbl\" (UniqueName: \"kubernetes.io/projected/eda1ddc3-ab20-471e-b273-9fa66556e35f-kube-api-access-ftkbl\") pod \"monitoring-plugin-77f4f76594-kff85\" (UID: \"eda1ddc3-ab20-471e-b273-9fa66556e35f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.880303 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880263    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-metrics-server-audit-profiles\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880375 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880328    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-client-ca-bundle\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880407 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880382    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-secret-metrics-server-client-certs\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880476 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880408    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880476 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880466    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7547x\" (UniqueName: \"kubernetes.io/projected/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-kube-api-access-7547x\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880538 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880490    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-secret-metrics-server-tls\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.880617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.880600    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eda1ddc3-ab20-471e-b273-9fa66556e35f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kff85\" (UID: \"eda1ddc3-ab20-471e-b273-9fa66556e35f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.882868 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.882846    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eda1ddc3-ab20-471e-b273-9fa66556e35f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kff85\" (UID: \"eda1ddc3-ab20-471e-b273-9fa66556e35f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.939834 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.939783    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftkbl\" (UniqueName: \"kubernetes.io/projected/eda1ddc3-ab20-471e-b273-9fa66556e35f-kube-api-access-ftkbl\") pod \"monitoring-plugin-77f4f76594-kff85\" (UID: \"eda1ddc3-ab20-471e-b273-9fa66556e35f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:05.981137 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981110    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-audit-log\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981151    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-metrics-server-audit-profiles\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981166    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-client-ca-bundle\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981186    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-secret-metrics-server-client-certs\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981202    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981218    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7547x\" (UniqueName: \"kubernetes.io/projected/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-kube-api-access-7547x\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981235    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-secret-metrics-server-tls\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981844 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981536    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-audit-log\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.981980 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.981962    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.982168 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.982146    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-metrics-server-audit-profiles\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.983711 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.983690    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-secret-metrics-server-client-certs\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.983761 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.983713    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-client-ca-bundle\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:05.983798 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:05.983780    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-secret-metrics-server-tls\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:06.007553 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.007531    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:06.038707 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.038687    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7547x\" (UniqueName: \"kubernetes.io/projected/2dca4577-7118-4c8c-a6f9-fb72f7b386ad-kube-api-access-7547x\") pod \"metrics-server-5d948c677c-pt8sl\" (UID: \"2dca4577-7118-4c8c-a6f9-fb72f7b386ad\") " pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:06.139057 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.139023    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:06.395797 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.395770    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kff85"]
Oct 30 23:42:06.398812 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:42:06.398781    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda1ddc3_ab20_471e_b273_9fa66556e35f.slice/crio-066d5252ee4450b5ea42e114449171b3a6f647394a5a1e1083819a234a1ac972 WatchSource:0}: Error finding container 066d5252ee4450b5ea42e114449171b3a6f647394a5a1e1083819a234a1ac972: Status 404 returned error can't find the container with id 066d5252ee4450b5ea42e114449171b3a6f647394a5a1e1083819a234a1ac972
Oct 30 23:42:06.528429 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.528358    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d948c677c-pt8sl"]
Oct 30 23:42:06.531058 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:42:06.531028    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dca4577_7118_4c8c_a6f9_fb72f7b386ad.slice/crio-53cb1bdf5d4e404aa0f8026664d8f0603ad332fba2cad6eef03d876345301b3a WatchSource:0}: Error finding container 53cb1bdf5d4e404aa0f8026664d8f0603ad332fba2cad6eef03d876345301b3a: Status 404 returned error can't find the container with id 53cb1bdf5d4e404aa0f8026664d8f0603ad332fba2cad6eef03d876345301b3a
Oct 30 23:42:06.980702 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.980663    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl" event={"ID":"2dca4577-7118-4c8c-a6f9-fb72f7b386ad","Type":"ContainerStarted","Data":"53cb1bdf5d4e404aa0f8026664d8f0603ad332fba2cad6eef03d876345301b3a"}
Oct 30 23:42:06.981753 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:06.981732    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85" event={"ID":"eda1ddc3-ab20-471e-b273-9fa66556e35f","Type":"ContainerStarted","Data":"066d5252ee4450b5ea42e114449171b3a6f647394a5a1e1083819a234a1ac972"}
Oct 30 23:42:07.921081 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:07.921049    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-568b89444d-55qld"
Oct 30 23:42:08.323018 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:42:08.322932    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:42:09.283724 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:09.283700    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t5xw7"
Oct 30 23:42:09.992820 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:09.992775    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl" event={"ID":"2dca4577-7118-4c8c-a6f9-fb72f7b386ad","Type":"ContainerStarted","Data":"6f4a5922013e03c20fd3fdc6a66fc1b6b234db345daf8811ee510726de2b9da7"}
Oct 30 23:42:09.994328 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:09.994298    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85" event={"ID":"eda1ddc3-ab20-471e-b273-9fa66556e35f","Type":"ContainerStarted","Data":"29bb47f1f119a587a0e38fdfc6c71389f08ee01edf3cb14c8e1c32648a9ceef5"}
Oct 30 23:42:09.994508 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:09.994491    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:09.999142 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:09.999121    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85"
Oct 30 23:42:10.111584 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:10.111534    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl" podStartSLOduration=2.452822073 podStartE2EDuration="5.111518673s" totalImagesPullingTime="2.6586966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:10.110258334 +0000 UTC m=+35.400806651" watchObservedRunningTime="2026-10-30 23:42:10.111518673 +0000 UTC m=+35.402066989"
Oct 30 23:42:10.219878 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:10.219825    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kff85" podStartSLOduration=2.430990244 podStartE2EDuration="5.21980902s" totalImagesPullingTime="2.788818776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:10.219114826 +0000 UTC m=+35.509663145" watchObservedRunningTime="2026-10-30 23:42:10.21980902 +0000 UTC m=+35.510357338"
Oct 30 23:42:17.907305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.907227    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64bcfc9dd8-n8ns5"]
Oct 30 23:42:17.910912 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.910895    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.969863 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.969830    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.969863 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.969869    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.970070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.969885    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.970070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.969959    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.970070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.970019    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.970172 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.970083    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:17.970172 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:17.970101    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqg9\" (UniqueName: \"kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.007129 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.007092    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 23:42:18.007129 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.007104    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l9lwt\""
Oct 30 23:42:18.007540 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.007092    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 23:42:18.008268 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.007713    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 23:42:18.008268 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.007900    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 23:42:18.008268 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.008120    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 23:42:18.008268 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.008154    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:42:18.008544 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.008388    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:42:18.011019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.010039    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bcfc9dd8-n8ns5"]
Oct 30 23:42:18.071321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071279    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071324    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqg9\" (UniqueName: \"kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071581 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071361    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071581 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071383    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071581 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071405    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071581 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071442    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071581 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071474    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.071932 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.071907    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.072086 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.072064    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.072198 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.072178    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.072256 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.072239    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.073954 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.073930    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.074067 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.073955    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.135661 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.135616    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqg9\" (UniqueName: \"kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9\") pod \"console-64bcfc9dd8-n8ns5\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") " pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.222749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.222668    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:18.589121 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:18.589024    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bcfc9dd8-n8ns5"]
Oct 30 23:42:18.591547 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:42:18.591521    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2acc91_f222_48c7_a5c3_f2ca21a088dc.slice/crio-a7118ea387939373bc70bf4711f07351f58cc119cbe2c22bf9d2c0995a1326ae WatchSource:0}: Error finding container a7118ea387939373bc70bf4711f07351f58cc119cbe2c22bf9d2c0995a1326ae: Status 404 returned error can't find the container with id a7118ea387939373bc70bf4711f07351f58cc119cbe2c22bf9d2c0995a1326ae
Oct 30 23:42:19.026396 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:19.026311    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcfc9dd8-n8ns5" event={"ID":"4b2acc91-f222-48c7-a5c3-f2ca21a088dc","Type":"ContainerStarted","Data":"a7118ea387939373bc70bf4711f07351f58cc119cbe2c22bf9d2c0995a1326ae"}
Oct 30 23:42:22.038866 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:22.038825    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcfc9dd8-n8ns5" event={"ID":"4b2acc91-f222-48c7-a5c3-f2ca21a088dc","Type":"ContainerStarted","Data":"2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e"}
Oct 30 23:42:22.150343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:22.150288    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64bcfc9dd8-n8ns5" podStartSLOduration=1.9147655829999999 podStartE2EDuration="5.150273336s" totalImagesPullingTime="3.235507753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:22.149943919 +0000 UTC m=+47.440492236" watchObservedRunningTime="2026-10-30 23:42:22.150273336 +0000 UTC m=+47.440821653"
Oct 30 23:42:26.139577 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:26.139535    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:26.140103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:26.139604    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:28.223037 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:28.222941    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:28.223426 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:28.223089    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:28.227624 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:28.227605    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:29.064765 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.064737    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:42:29.389046 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.389012    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdbdc75bf-zvjcw"]
Oct 30 23:42:29.394088 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.394070    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.459934 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.459903    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460089 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.459942    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460089 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.460072    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460168 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.460113    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460168 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.460139    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjp8\" (UniqueName: \"kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460168 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.460159    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460258 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.460229    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.460297 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.460273    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.536924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.536884    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 23:42:29.537914 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.537895    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdbdc75bf-zvjcw"]
Oct 30 23:42:29.561362 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561331    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjp8\" (UniqueName: \"kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561459 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561371    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561459 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561398    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561459 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561423    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561459 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561447    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561595 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561472    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561730 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561707    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.561828 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.561766    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.562238 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.562173    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.562352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.562305    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.562352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.562331    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.562476 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.562349    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.562476 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.562415    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.563942 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.563923    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.564022 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.563953    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.619442 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.619408    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjp8\" (UniqueName: \"kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8\") pod \"console-5fdbdc75bf-zvjcw\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") " pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:29.706486 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:29.706387    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:30.076911 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:30.076829    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdbdc75bf-zvjcw"]
Oct 30 23:42:30.079921 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:42:30.079895    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d37b972_f18f_48b1_8323_f03e54a26e90.slice/crio-e124052751143f0135c9fa5e3901c816323d61d2ec5f062318e17037174d6e5a WatchSource:0}: Error finding container e124052751143f0135c9fa5e3901c816323d61d2ec5f062318e17037174d6e5a: Status 404 returned error can't find the container with id e124052751143f0135c9fa5e3901c816323d61d2ec5f062318e17037174d6e5a
Oct 30 23:42:31.069070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:31.069034    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdbdc75bf-zvjcw" event={"ID":"1d37b972-f18f-48b1-8323-f03e54a26e90","Type":"ContainerStarted","Data":"8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d"}
Oct 30 23:42:31.069460 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:31.069086    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdbdc75bf-zvjcw" event={"ID":"1d37b972-f18f-48b1-8323-f03e54a26e90","Type":"ContainerStarted","Data":"e124052751143f0135c9fa5e3901c816323d61d2ec5f062318e17037174d6e5a"}
Oct 30 23:42:31.180966 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:31.180866    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fdbdc75bf-zvjcw" podStartSLOduration=2.18084891 podStartE2EDuration="2.18084891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:31.18059509 +0000 UTC m=+56.471143419" watchObservedRunningTime="2026-10-30 23:42:31.18084891 +0000 UTC m=+56.471397229"
Oct 30 23:42:38.317345 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:42:38.317299    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:42:39.707176 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:39.707116    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:39.707715 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:39.707233    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:39.711959 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:39.711934    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:40.105111 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:40.105086    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:42:40.245881 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:40.245852    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bcfc9dd8-n8ns5"]
Oct 30 23:42:46.145097 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:46.145068    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:46.148688 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:46.148662    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d948c677c-pt8sl"
Oct 30 23:42:50.153576 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.153544    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-794c64c68-5s7zk"]
Oct 30 23:42:50.156730 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.156713    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.254207 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.254176    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794c64c68-5s7zk"]
Oct 30 23:42:50.324902 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.324865    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-config\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.324902 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.324902    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-service-ca\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.325113 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.324920    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-kube-api-access-rtw84\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.325113 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.325001    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-oauth-config\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.325113 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.325048    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-oauth-serving-cert\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.325113 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.325093    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-tmp\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.325234 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.325142    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-serving-cert\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.325234 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.325201    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-trusted-ca-bundle\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.425832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425760    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-serving-cert\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.425832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425806    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-trusted-ca-bundle\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426003 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425836    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-config\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426003 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425851    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-service-ca\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426003 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425867    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-kube-api-access-rtw84\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426003 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425891    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-oauth-config\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426003 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.425912    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-oauth-serving-cert\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426345 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.426316    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-tmp\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.426679    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-service-ca\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426794 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.426680    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-tmp\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426794 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.426713    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-config\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426875 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.426823    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-trusted-ca-bundle\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.426875 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.426827    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-oauth-serving-cert\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.428415 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.428389    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-serving-cert\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.428659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.428625    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-console-oauth-config\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.486123 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.486084    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/043e8e5f-e8ac-40a2-8fa2-4cd4144fd691-kube-api-access-rtw84\") pod \"console-794c64c68-5s7zk\" (UID: \"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691\") " pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:50.767678 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:50.767580    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:42:51.134127 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:51.134095    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794c64c68-5s7zk"]
Oct 30 23:42:51.162734 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:42:51.162704    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043e8e5f_e8ac_40a2_8fa2_4cd4144fd691.slice/crio-3f08f6453f76f3ad6f045232d65931d7dd19421ef612baa72d12a587406dae53 WatchSource:0}: Error finding container 3f08f6453f76f3ad6f045232d65931d7dd19421ef612baa72d12a587406dae53: Status 404 returned error can't find the container with id 3f08f6453f76f3ad6f045232d65931d7dd19421ef612baa72d12a587406dae53
Oct 30 23:42:52.140743 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:52.140706    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794c64c68-5s7zk" event={"ID":"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691","Type":"ContainerStarted","Data":"e6505d2420450252d21636330071de822f8ffc34ec7f7347f380331b0c9b6329"}
Oct 30 23:42:52.140929 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:52.140757    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794c64c68-5s7zk" event={"ID":"043e8e5f-e8ac-40a2-8fa2-4cd4144fd691","Type":"ContainerStarted","Data":"3f08f6453f76f3ad6f045232d65931d7dd19421ef612baa72d12a587406dae53"}
Oct 30 23:42:52.252874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:42:52.252821    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-794c64c68-5s7zk" podStartSLOduration=2.252806054 podStartE2EDuration="2.252806054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:52.252209198 +0000 UTC m=+77.542757528" watchObservedRunningTime="2026-10-30 23:42:52.252806054 +0000 UTC m=+77.543354372"
Oct 30 23:43:00.768334 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:00.768284    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:43:00.768747 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:00.768355    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:43:00.773122 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:00.773101    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:43:01.174808 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:01.174781    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794c64c68-5s7zk"
Oct 30 23:43:01.335319 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:01.335285    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdbdc75bf-zvjcw"]
Oct 30 23:43:05.264561 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.264513    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64bcfc9dd8-n8ns5" podUID="4b2acc91-f222-48c7-a5c3-f2ca21a088dc" containerName="console" containerID="cri-o://2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e" gracePeriod=15
Oct 30 23:43:05.511857 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.511832    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:43:05.649314 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649280    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649314 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649319    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649350    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649489    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9\" (UniqueName: \"kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649603 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649540    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649603 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649592    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649713 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649618    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config\") pod \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\" (UID: \"4b2acc91-f222-48c7-a5c3-f2ca21a088dc\") "
Oct 30 23:43:05.649713 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649681    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:05.649907 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649879    4466 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-tmp\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:05.649979 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649927    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:05.650034 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.649988    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:05.650081 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.650050    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:05.651702 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.651670    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "kube-api-access-shqg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:05.651956 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.651935    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:05.651956 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.651950    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert" pod "4b2acc91-f222-48c7-a5c3-f2ca21a088dc" (UID: "4b2acc91-f222-48c7-a5c3-f2ca21a088dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:05.750343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.750285    4466 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-serving-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:05.750343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.750335    4466 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-oauth-config\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:05.750343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.750345    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqg9\" (UniqueName: \"kubernetes.io/projected/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-kube-api-access-shqg9\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:05.750343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.750357    4466 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:05.750343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.750366    4466 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-oauth-serving-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:05.750666 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:05.750374    4466 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b2acc91-f222-48c7-a5c3-f2ca21a088dc-console-config\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:06.186404 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.186368    4466 generic.go:356] "Generic (PLEG): container finished" podID="4b2acc91-f222-48c7-a5c3-f2ca21a088dc" containerID="2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e" exitCode=0
Oct 30 23:43:06.186602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.186415    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bcfc9dd8-n8ns5"
Oct 30 23:43:06.186602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.186437    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcfc9dd8-n8ns5" event={"ID":"4b2acc91-f222-48c7-a5c3-f2ca21a088dc","Type":"ContainerDied","Data":"2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e"}
Oct 30 23:43:06.186602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.186484    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bcfc9dd8-n8ns5" event={"ID":"4b2acc91-f222-48c7-a5c3-f2ca21a088dc","Type":"ContainerDied","Data":"a7118ea387939373bc70bf4711f07351f58cc119cbe2c22bf9d2c0995a1326ae"}
Oct 30 23:43:06.186602 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.186508    4466 scope.go:122] "RemoveContainer" containerID="2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e"
Oct 30 23:43:06.194697 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.194673    4466 scope.go:122] "RemoveContainer" containerID="2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e"
Oct 30 23:43:06.194981 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:43:06.194945    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e\": container with ID starting with 2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e not found: ID does not exist" containerID="2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e"
Oct 30 23:43:06.195031 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.194990    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e"} err="failed to get container status \"2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e\": rpc error: code = NotFound desc = could not find container \"2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e\": container with ID starting with 2c5d478748678d62e348a2d591c1483a4f11b934ceef6e3a61940ea8b895976e not found: ID does not exist"
Oct 30 23:43:06.341209 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.341173    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bcfc9dd8-n8ns5"]
Oct 30 23:43:06.345080 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.345052    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64bcfc9dd8-n8ns5"]
Oct 30 23:43:06.824956 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:06.824923    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2acc91-f222-48c7-a5c3-f2ca21a088dc" path="/var/lib/kubelet/pods/4b2acc91-f222-48c7-a5c3-f2ca21a088dc/volumes"
Oct 30 23:43:08.314352 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:43:08.314308    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:43:26.353823 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.353783    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fdbdc75bf-zvjcw" podUID="1d37b972-f18f-48b1-8323-f03e54a26e90" containerName="console" containerID="cri-o://8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d" gracePeriod=15
Oct 30 23:43:26.601404 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.601379    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:43:26.691908 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691827    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8\" (UniqueName: \"kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.691908 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691873    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.691908 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691903    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.692184 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691929    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.692184 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691952    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.692184 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691968    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.692184 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.691986    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.692184 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.692047    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp\") pod \"1d37b972-f18f-48b1-8323-f03e54a26e90\" (UID: \"1d37b972-f18f-48b1-8323-f03e54a26e90\") "
Oct 30 23:43:26.692456 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.692420    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:26.692515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.692354    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:26.692515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.692436    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:26.692515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.692448    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:26.692686 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.692628    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:26.694059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.694033    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "kube-api-access-qrjp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:26.694172 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.694056    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:26.694211 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.694170    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config" pod "1d37b972-f18f-48b1-8323-f03e54a26e90" (UID: "1d37b972-f18f-48b1-8323-f03e54a26e90"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:26.793104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793055    4466 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-serving-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793098    4466 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-oauth-serving-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793108    4466 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-console-config\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793117    4466 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d37b972-f18f-48b1-8323-f03e54a26e90-console-oauth-config\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793126    4466 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-trusted-ca-bundle\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793444 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793134    4466 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d37b972-f18f-48b1-8323-f03e54a26e90-tmp\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793444 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793144    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjp8\" (UniqueName: \"kubernetes.io/projected/1d37b972-f18f-48b1-8323-f03e54a26e90-kube-api-access-qrjp8\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:26.793444 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:26.793154    4466 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d37b972-f18f-48b1-8323-f03e54a26e90-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:27.254356 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.254321    4466 generic.go:356] "Generic (PLEG): container finished" podID="1d37b972-f18f-48b1-8323-f03e54a26e90" containerID="8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d" exitCode=0
Oct 30 23:43:27.254521 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.254371    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdbdc75bf-zvjcw"
Oct 30 23:43:27.254521 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.254399    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdbdc75bf-zvjcw" event={"ID":"1d37b972-f18f-48b1-8323-f03e54a26e90","Type":"ContainerDied","Data":"8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d"}
Oct 30 23:43:27.254521 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.254448    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdbdc75bf-zvjcw" event={"ID":"1d37b972-f18f-48b1-8323-f03e54a26e90","Type":"ContainerDied","Data":"e124052751143f0135c9fa5e3901c816323d61d2ec5f062318e17037174d6e5a"}
Oct 30 23:43:27.254521 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.254469    4466 scope.go:122] "RemoveContainer" containerID="8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d"
Oct 30 23:43:27.261889 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.261869    4466 scope.go:122] "RemoveContainer" containerID="8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d"
Oct 30 23:43:27.262192 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:43:27.262169    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d\": container with ID starting with 8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d not found: ID does not exist" containerID="8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d"
Oct 30 23:43:27.262270 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.262201    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d"} err="failed to get container status \"8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d\": rpc error: code = NotFound desc = could not find container \"8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d\": container with ID starting with 8bd7dd96ae1b9c376c8454209e990e165e8db38a7ea767db96d84d46cb6ad94d not found: ID does not exist"
Oct 30 23:43:27.409034 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.409001    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdbdc75bf-zvjcw"]
Oct 30 23:43:27.455656 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:27.455599    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdbdc75bf-zvjcw"]
Oct 30 23:43:28.824455 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:43:28.824380    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d37b972-f18f-48b1-8323-f03e54a26e90" path="/var/lib/kubelet/pods/1d37b972-f18f-48b1-8323-f03e54a26e90/volumes"
Oct 30 23:43:38.314669 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:43:38.314609    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:44:08.314901 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:44:08.314865    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:44:38.315207 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:44:38.315171    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:45:04.145270 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145234    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"]
Oct 30 23:45:04.145792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145483    4466 cpu_manager.go:414] "containerMap: removing container" podUID="4b2acc91-f222-48c7-a5c3-f2ca21a088dc" containerName="console"
Oct 30 23:45:04.145792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145495    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2acc91-f222-48c7-a5c3-f2ca21a088dc" containerName="console"
Oct 30 23:45:04.145792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145516    4466 cpu_manager.go:414] "containerMap: removing container" podUID="1d37b972-f18f-48b1-8323-f03e54a26e90" containerName="console"
Oct 30 23:45:04.145792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145520    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d37b972-f18f-48b1-8323-f03e54a26e90" containerName="console"
Oct 30 23:45:04.145792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145560    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2acc91-f222-48c7-a5c3-f2ca21a088dc" containerName="console"
Oct 30 23:45:04.145792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.145570    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d37b972-f18f-48b1-8323-f03e54a26e90" containerName="console"
Oct 30 23:45:04.148651 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.148605    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.195920 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.195881    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.195920 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.195918    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.196199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.195935    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.196199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.195952    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.196199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.196019    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.196199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.196124    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.196363 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.196203    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hw5l\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.199836 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.199803    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 23:45:04.201436 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.201417    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 23:45:04.201608 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.201596    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:45:04.246908 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.246874    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 23:45:04.247085 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.246933    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 23:45:04.249170 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.249147    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"]
Oct 30 23:45:04.297680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297615    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.297680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297676    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.297946 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297696    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.297946 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297711    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.297946 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297733    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.297946 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297751    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.297946 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.297823    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hw5l\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.298456 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.298434    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.300131 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.300108    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 23:45:04.300131 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.300108    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbdd5\""
Oct 30 23:45:04.300290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.300185    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.300290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.300191    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.300290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.300248    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.310257 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.310227    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.356125 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.356086    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.356406 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.356382    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hw5l\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l\") pod \"istiod-openshift-gateway-7876bfcb74-x544h\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.459449 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.459361    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:04.873696 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:04.873672    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"]
Oct 30 23:45:04.875885 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:45:04.875856    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4171e8b3_5333_490a_ba8b_aa39f848a7f4.slice/crio-0d6a3b20cf9946624e78075cb1d76d90adcb90712a8d4d41fdca7322e616c18d WatchSource:0}: Error finding container 0d6a3b20cf9946624e78075cb1d76d90adcb90712a8d4d41fdca7322e616c18d: Status 404 returned error can't find the container with id 0d6a3b20cf9946624e78075cb1d76d90adcb90712a8d4d41fdca7322e616c18d
Oct 30 23:45:05.558017 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:05.557975    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" event={"ID":"4171e8b3-5333-490a-ba8b-aa39f848a7f4","Type":"ContainerStarted","Data":"0d6a3b20cf9946624e78075cb1d76d90adcb90712a8d4d41fdca7322e616c18d"}
Oct 30 23:45:08.317267 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:45:08.317229    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:45:09.349363 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:09.349327    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:45:09.349676 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:09.349393    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:45:09.572886 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:09.572851    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" event={"ID":"4171e8b3-5333-490a-ba8b-aa39f848a7f4","Type":"ContainerStarted","Data":"7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597"}
Oct 30 23:45:09.573069 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:09.572991    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:09.733846 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:09.733739    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" podStartSLOduration=1.262832387 podStartE2EDuration="5.733720614s" totalImagesPullingTime="4.470888227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:09.733291371 +0000 UTC m=+215.023839686" watchObservedRunningTime="2026-10-30 23:45:09.733720614 +0000 UTC m=+215.024268933"
Oct 30 23:45:10.580615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:10.580555    4466 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-x544h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:45:10.581082 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:10.580667    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" podUID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:45:13.576876 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:13.576842    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:45:38.315498 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:45:38.315459    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:45:53.935119 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.935082    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"]
Oct 30 23:45:53.938165 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.938147    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:53.974434 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.974401    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08738562-1fd1-48f2-b436-d536a94ccc69-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:53.974599 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.974445    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:53.974599 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.974502    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:53.974599 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.974553    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffnf\" (UniqueName: \"kubernetes.io/projected/08738562-1fd1-48f2-b436-d536a94ccc69-kube-api-access-xffnf\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:53.974776 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.974628    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:53.974776 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:53.974669    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08738562-1fd1-48f2-b436-d536a94ccc69-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.012119 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.012084    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 23:45:54.012119 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.012103    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 23:45:54.042474 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.042442    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"]
Oct 30 23:45:54.075120 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075094    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08738562-1fd1-48f2-b436-d536a94ccc69-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.075307 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075132    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.075307 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075158    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.075307 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075184    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffnf\" (UniqueName: \"kubernetes.io/projected/08738562-1fd1-48f2-b436-d536a94ccc69-kube-api-access-xffnf\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.075307 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075216    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.075307 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075241    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08738562-1fd1-48f2-b436-d536a94ccc69-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.075583 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.075553    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08738562-1fd1-48f2-b436-d536a94ccc69-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.077352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.077334    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08738562-1fd1-48f2-b436-d536a94ccc69-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.087887 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.087860    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pxfm6\""
Oct 30 23:45:54.087887 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.087861    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 23:45:54.088060 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.087898    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 23:45:54.088060 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.087946    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 23:45:54.098095 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.098062    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.098224 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.098122    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.098224 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.098171    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08738562-1fd1-48f2-b436-d536a94ccc69-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.138106 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.138070    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffnf\" (UniqueName: \"kubernetes.io/projected/08738562-1fd1-48f2-b436-d536a94ccc69-kube-api-access-xffnf\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-6qt75\" (UID: \"08738562-1fd1-48f2-b436-d536a94ccc69\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.253351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.253169    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:45:54.737402 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:54.737303    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"]
Oct 30 23:45:54.739948 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:45:54.739920    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08738562_1fd1_48f2_b436_d536a94ccc69.slice/crio-deda979b78431ad3aabb6a6e10ced4aa7804180e9d21875c9e73c70605654e8c WatchSource:0}: Error finding container deda979b78431ad3aabb6a6e10ced4aa7804180e9d21875c9e73c70605654e8c: Status 404 returned error can't find the container with id deda979b78431ad3aabb6a6e10ced4aa7804180e9d21875c9e73c70605654e8c
Oct 30 23:45:55.716414 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:55.716378    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75" event={"ID":"08738562-1fd1-48f2-b436-d536a94ccc69","Type":"ContainerStarted","Data":"deda979b78431ad3aabb6a6e10ced4aa7804180e9d21875c9e73c70605654e8c"}
Oct 30 23:45:58.728818 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:58.728779    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75" event={"ID":"08738562-1fd1-48f2-b436-d536a94ccc69","Type":"ContainerStarted","Data":"55d5046e668745141294a15c9607d8a3124b94b1cb98155462c9ab65ae70d622"}
Oct 30 23:45:59.733738 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:59.733697    4466 generic.go:356] "Generic (PLEG): container finished" podID="08738562-1fd1-48f2-b436-d536a94ccc69" containerID="55d5046e668745141294a15c9607d8a3124b94b1cb98155462c9ab65ae70d622" exitCode=0
Oct 30 23:45:59.734159 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:45:59.733776    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75" event={"ID":"08738562-1fd1-48f2-b436-d536a94ccc69","Type":"ContainerDied","Data":"55d5046e668745141294a15c9607d8a3124b94b1cb98155462c9ab65ae70d622"}
Oct 30 23:46:00.739029 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:00.738994    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75" event={"ID":"08738562-1fd1-48f2-b436-d536a94ccc69","Type":"ContainerStarted","Data":"912568e1e7ac07f39c508523a91243543378f7037658531d4efc87bb52e72db6"}
Oct 30 23:46:00.739447 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:00.739147    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:46:00.967144 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:00.967045    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75" podStartSLOduration=3.170022016 podStartE2EDuration="7.967026727s" totalImagesPullingTime="4.327761381s" totalInitContainerRuntime="469.24333ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:00.920417461 +0000 UTC m=+266.210965783" watchObservedRunningTime="2026-10-30 23:46:00.967026727 +0000 UTC m=+266.257575044"
Oct 30 23:46:08.316487 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:46:08.316445    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:46:11.750370 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:11.750335    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-6qt75"
Oct 30 23:46:29.869924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.869891    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"]
Oct 30 23:46:29.871830 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.871815    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:29.923451 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.923422    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 23:46:29.947912 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.947867    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986a67d7-902f-452a-a839-391beaa99b68-tls-certs\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:29.948105 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.947924    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986a67d7-902f-452a-a839-391beaa99b68-tmp\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:29.948105 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.948019    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hl2\" (UniqueName: \"kubernetes.io/projected/986a67d7-902f-452a-a839-391beaa99b68-kube-api-access-89hl2\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:29.967670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.967628    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 23:46:29.967670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.967629    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q8vk7\""
Oct 30 23:46:29.968744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:29.968722    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"]
Oct 30 23:46:30.049388 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.049354    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hl2\" (UniqueName: \"kubernetes.io/projected/986a67d7-902f-452a-a839-391beaa99b68-kube-api-access-89hl2\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.049562 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.049419    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986a67d7-902f-452a-a839-391beaa99b68-tls-certs\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.049562 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.049453    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986a67d7-902f-452a-a839-391beaa99b68-tmp\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.051872 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.051845    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986a67d7-902f-452a-a839-391beaa99b68-tmp\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.051979 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.051952    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986a67d7-902f-452a-a839-391beaa99b68-tls-certs\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.116928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.116897    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hl2\" (UniqueName: \"kubernetes.io/projected/986a67d7-902f-452a-a839-391beaa99b68-kube-api-access-89hl2\") pod \"kube-auth-proxy-546bf7cb95-4gjhl\" (UID: \"986a67d7-902f-452a-a839-391beaa99b68\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.182426 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.182346    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"
Oct 30 23:46:30.544814 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.544726    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl"]
Oct 30 23:46:30.548388 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:46:30.548359    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986a67d7_902f_452a_a839_391beaa99b68.slice/crio-93b5ab8d065df0f57645a554b2e22ded359e3880b9df10714896f898e7f82156 WatchSource:0}: Error finding container 93b5ab8d065df0f57645a554b2e22ded359e3880b9df10714896f898e7f82156: Status 404 returned error can't find the container with id 93b5ab8d065df0f57645a554b2e22ded359e3880b9df10714896f898e7f82156
Oct 30 23:46:30.839547 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:30.839522    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl" event={"ID":"986a67d7-902f-452a-a839-391beaa99b68","Type":"ContainerStarted","Data":"93b5ab8d065df0f57645a554b2e22ded359e3880b9df10714896f898e7f82156"}
Oct 30 23:46:34.854043 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:34.854008    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl" event={"ID":"986a67d7-902f-452a-a839-391beaa99b68","Type":"ContainerStarted","Data":"94984698f790c4bdbab288b97ef64abaa1f220e4da8b2519fbef87377b3ba690"}
Oct 30 23:46:34.966530 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:34.966472    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-4gjhl" podStartSLOduration=2.445173243 podStartE2EDuration="5.966456949s" totalImagesPullingTime="3.521283706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:34.965086093 +0000 UTC m=+300.255634468" watchObservedRunningTime="2026-10-30 23:46:34.966456949 +0000 UTC m=+300.257005267"
Oct 30 23:46:38.315011 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:46:38.314973    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:46:52.750705 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.750610    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"]
Oct 30 23:46:52.753881 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.753860    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.835499 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.835467    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-manager-config\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.835675 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.835521    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4pr\" (UniqueName: \"kubernetes.io/projected/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-kube-api-access-tm4pr\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.835675 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.835628    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-metrics-certs\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.835675 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.835670    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-cert\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.851213 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.851182    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:46:52.851276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.851228    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 23:46:52.851276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.851252    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 23:46:52.851276 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.851263    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-nmfzk\""
Oct 30 23:46:52.852010 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.851991    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 23:46:52.852010 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.851990    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 23:46:52.893673 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.893626    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"]
Oct 30 23:46:52.936295 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.936258    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4pr\" (UniqueName: \"kubernetes.io/projected/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-kube-api-access-tm4pr\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.936485 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.936329    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-metrics-certs\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.936485 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.936348    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-cert\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.936485 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.936388    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-manager-config\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.937174 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.937148    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-manager-config\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.938825 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.938807    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-cert\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.938927 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.938907    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-metrics-certs\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:52.998570 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:52.998539    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4pr\" (UniqueName: \"kubernetes.io/projected/6eba02cd-e7bd-4632-87b5-eb29bf4556bb-kube-api-access-tm4pr\") pod \"jobset-controller-manager-645dd65f76-hz96m\" (UID: \"6eba02cd-e7bd-4632-87b5-eb29bf4556bb\") " pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:53.065622 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:53.065536    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:53.433402 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:53.433369    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"]
Oct 30 23:46:53.437080 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:46:53.437051    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eba02cd_e7bd_4632_87b5_eb29bf4556bb.slice/crio-13ff55ec2e8433431fcc396bc2ca15691739b82e4087dd6a00be796b57f6ede0 WatchSource:0}: Error finding container 13ff55ec2e8433431fcc396bc2ca15691739b82e4087dd6a00be796b57f6ede0: Status 404 returned error can't find the container with id 13ff55ec2e8433431fcc396bc2ca15691739b82e4087dd6a00be796b57f6ede0
Oct 30 23:46:53.438958 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:53.438941    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:46:53.917073 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:53.917038    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m" event={"ID":"6eba02cd-e7bd-4632-87b5-eb29bf4556bb","Type":"ContainerStarted","Data":"13ff55ec2e8433431fcc396bc2ca15691739b82e4087dd6a00be796b57f6ede0"}
Oct 30 23:46:58.937879 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:58.937844    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m" event={"ID":"6eba02cd-e7bd-4632-87b5-eb29bf4556bb","Type":"ContainerStarted","Data":"81cfb866abe5137442e9d3145987fd4a6b5e42298f3e593ece7988c550e56063"}
Oct 30 23:46:58.938293 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:58.937986    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:46:59.052427 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:46:59.052371    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m" podStartSLOduration=2.295345651 podStartE2EDuration="7.052354935s" totalImagesPullingTime="4.757009284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:59.050961784 +0000 UTC m=+324.341510093" watchObservedRunningTime="2026-10-30 23:46:59.052354935 +0000 UTC m=+324.342903247"
Oct 30 23:47:08.315871 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:47:08.315828    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:47:09.946745 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:09.946717    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-645dd65f76-hz96m"
Oct 30 23:47:38.316114 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:47:38.316071    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:47:52.631616 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.631579    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"]
Oct 30 23:47:52.632034 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.631801    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" podUID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" containerName="discovery" containerID="cri-o://7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597" gracePeriod=30
Oct 30 23:47:52.864950 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.864921    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:47:52.932323 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932247    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932323 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932306    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932528 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932331    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932528 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932362    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932528 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932388    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932528 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932476    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932757 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932538    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig\") pod \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\" (UID: \"4171e8b3-5333-490a-ba8b-aa39f848a7f4\") "
Oct 30 23:47:52.932757 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932726    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:47:52.932920 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.932897    4466 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-ca-configmap\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:52.935283 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.935244    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:52.935430 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.935304    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:52.935430 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.935325    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:52.935430 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.935333    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:52.935430 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.935348    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "kube-api-access-6hw5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:52.935566 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:52.935460    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token" pod "4171e8b3-5333-490a-ba8b-aa39f848a7f4" (UID: "4171e8b3-5333-490a-ba8b-aa39f848a7f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:53.034084 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.034030    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.034084 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.034074    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hw5l\" (UniqueName: \"kubernetes.io/projected/4171e8b3-5333-490a-ba8b-aa39f848a7f4-kube-api-access-6hw5l\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.034084 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.034085    4466 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4171e8b3-5333-490a-ba8b-aa39f848a7f4-local-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.034084 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.034093    4466 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-kubeconfig\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.034084 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.034103    4466 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-istio-csr-dns-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.034386 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.034111    4466 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4171e8b3-5333-490a-ba8b-aa39f848a7f4-cacerts\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.138181 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.138147    4466 generic.go:356] "Generic (PLEG): container finished" podID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" containerID="7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597" exitCode=0
Oct 30 23:47:53.138351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.138189    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" event={"ID":"4171e8b3-5333-490a-ba8b-aa39f848a7f4","Type":"ContainerDied","Data":"7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597"}
Oct 30 23:47:53.138351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.138193    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"
Oct 30 23:47:53.138351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.138234    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h" event={"ID":"4171e8b3-5333-490a-ba8b-aa39f848a7f4","Type":"ContainerDied","Data":"0d6a3b20cf9946624e78075cb1d76d90adcb90712a8d4d41fdca7322e616c18d"}
Oct 30 23:47:53.138351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.138256    4466 scope.go:122] "RemoveContainer" containerID="7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597"
Oct 30 23:47:53.149729 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.149709    4466 scope.go:122] "RemoveContainer" containerID="7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597"
Oct 30 23:47:53.150029 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:47:53.150012    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597\": container with ID starting with 7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597 not found: ID does not exist" containerID="7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597"
Oct 30 23:47:53.150087 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.150037    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597"} err="failed to get container status \"7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597\": rpc error: code = NotFound desc = could not find container \"7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597\": container with ID starting with 7aee655d735a89370b7b9aa28849cd425c9761e4fde9573cc64f09cae2c49597 not found: ID does not exist"
Oct 30 23:47:53.296501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.296428    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"]
Oct 30 23:47:53.298815 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:53.298790    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x544h"]
Oct 30 23:47:54.824688 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:54.824658    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" path="/var/lib/kubelet/pods/4171e8b3-5333-490a-ba8b-aa39f848a7f4/volumes"
Oct 30 23:47:56.626762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.626731    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"]
Oct 30 23:47:56.627150 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.627002    4466 cpu_manager.go:414] "containerMap: removing container" podUID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" containerName="discovery"
Oct 30 23:47:56.627150 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.627012    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" containerName="discovery"
Oct 30 23:47:56.627150 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.627062    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="4171e8b3-5333-490a-ba8b-aa39f848a7f4" containerName="discovery"
Oct 30 23:47:56.631161 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.631142    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.727189 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.727144    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 23:47:56.727189 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.727202    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nk58\""
Oct 30 23:47:56.727189 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.727208    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:47:56.727532 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.727232    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:47:56.727532 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.727250    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 23:47:56.729764 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.729738    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"]
Oct 30 23:47:56.765854 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.765805    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.765854 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.765862    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzpf\" (UniqueName: \"kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.766054 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.765883    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.766054 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.766018    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.866976 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.866940    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.867149 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.867006    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.867149 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.867039    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzpf\" (UniqueName: \"kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.867149 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.867066    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.867744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.867722    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.869452 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.869430    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.869550 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.869533    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.924566 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.924491    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzpf\" (UniqueName: \"kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf\") pod \"maas-api-6d669c7fcd-fpbjh\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:56.943408 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:56.943372    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:47:57.337423 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:57.337394    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"]
Oct 30 23:47:57.340440 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:47:57.340409    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e8e1c9_1b6d_40b0_9f18_d2dff9795d93.slice/crio-f60576746f6611e26cdbfaa3107b225074a29b06abdd60fea142e808fe93d549 WatchSource:0}: Error finding container f60576746f6611e26cdbfaa3107b225074a29b06abdd60fea142e808fe93d549: Status 404 returned error can't find the container with id f60576746f6611e26cdbfaa3107b225074a29b06abdd60fea142e808fe93d549
Oct 30 23:47:58.157025 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:47:58.156988    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh" event={"ID":"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93","Type":"ContainerStarted","Data":"f60576746f6611e26cdbfaa3107b225074a29b06abdd60fea142e808fe93d549"}
Oct 30 23:48:01.169336 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:01.169299    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh" event={"ID":"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93","Type":"ContainerStarted","Data":"58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5"}
Oct 30 23:48:01.169782 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:01.169420    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:48:01.281922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:01.281806    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh" podStartSLOduration=1.6920236100000001 podStartE2EDuration="5.281782256s" totalImagesPullingTime="3.589758646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:01.281283456 +0000 UTC m=+386.571831776" watchObservedRunningTime="2026-10-30 23:48:01.281782256 +0000 UTC m=+386.572330574"
Oct 30 23:48:07.179017 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:07.178989    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:48:08.252205 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.252168    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"]
Oct 30 23:48:08.258485 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.258463    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.317058 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:48:08.317024    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:48:08.355505 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.355469    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"]
Oct 30 23:48:08.364856 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.364830    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.364983 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.364889    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.365024 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.364976    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.365024 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.365013    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcd9q\" (UniqueName: \"kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.466090 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.466058    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.466090 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.466097    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.466352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.466122    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcd9q\" (UniqueName: \"kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.466352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.466164    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.466833 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.466800    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.468580 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.468557    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.468685 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.468602    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.523860 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.523786    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcd9q\" (UniqueName: \"kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q\") pod \"maas-api-56bf677d9d-kn27r\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") " pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.571529 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.571494    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:08.937768 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:08.937731    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"]
Oct 30 23:48:08.940904 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:48:08.940881    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653ca4d3_0a74_45c5_b15a_f5e2076fa656.slice/crio-fc02ad1d0270c58b997b967d41c0cce45461439e8baee4a0f319b1391f60ab6a WatchSource:0}: Error finding container fc02ad1d0270c58b997b967d41c0cce45461439e8baee4a0f319b1391f60ab6a: Status 404 returned error can't find the container with id fc02ad1d0270c58b997b967d41c0cce45461439e8baee4a0f319b1391f60ab6a
Oct 30 23:48:09.200421 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:09.200328    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" event={"ID":"653ca4d3-0a74-45c5-b15a-f5e2076fa656","Type":"ContainerStarted","Data":"fc02ad1d0270c58b997b967d41c0cce45461439e8baee4a0f319b1391f60ab6a"}
Oct 30 23:48:10.205859 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:10.205768    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" event={"ID":"653ca4d3-0a74-45c5-b15a-f5e2076fa656","Type":"ContainerStarted","Data":"b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77"}
Oct 30 23:48:10.206242 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:10.205864    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:10.318388 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:10.318326    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" podStartSLOduration=1.382768558 podStartE2EDuration="2.318312311s" totalImagesPullingTime="935.543753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:10.317025757 +0000 UTC m=+395.607574075" watchObservedRunningTime="2026-10-30 23:48:10.318312311 +0000 UTC m=+395.608860628"
Oct 30 23:48:16.214520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.214491    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 30 23:48:16.374313 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.374243    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"]
Oct 30 23:48:16.374478 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.374457    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh" podUID="e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" containerName="maas-api" containerID="cri-o://58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5" gracePeriod=30
Oct 30 23:48:16.610674 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.610648    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:48:16.623228 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.623194    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf\" (UniqueName: \"kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf\") pod \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") "
Oct 30 23:48:16.623396 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.623243    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls\") pod \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") "
Oct 30 23:48:16.623396 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.623306    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca\") pod \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") "
Oct 30 23:48:16.623396 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.623370    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs\") pod \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\" (UID: \"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93\") "
Oct 30 23:48:16.623725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.623694    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca" pod "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" (UID: "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:16.625520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.625493    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls" pod "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" (UID: "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:16.625520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.625506    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs" pod "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" (UID: "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:16.625716 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.625546    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf" pod "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" (UID: "e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93"). InnerVolumeSpecName "kube-api-access-7dzpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:16.723928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.723895    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzpf\" (UniqueName: \"kubernetes.io/projected/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-kube-api-access-7dzpf\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:16.723928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.723926    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:16.723928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.723936    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:16.724192 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:16.723945    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:17.232984 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.232947    4466 generic.go:356] "Generic (PLEG): container finished" podID="e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" containerID="58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5" exitCode=0
Oct 30 23:48:17.233437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.232994    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh" event={"ID":"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93","Type":"ContainerDied","Data":"58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5"}
Oct 30 23:48:17.233437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.233000    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"
Oct 30 23:48:17.233437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.233045    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh" event={"ID":"e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93","Type":"ContainerDied","Data":"f60576746f6611e26cdbfaa3107b225074a29b06abdd60fea142e808fe93d549"}
Oct 30 23:48:17.233437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.233073    4466 scope.go:122] "RemoveContainer" containerID="58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5"
Oct 30 23:48:17.241137 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.241113    4466 scope.go:122] "RemoveContainer" containerID="58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5"
Oct 30 23:48:17.241408 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:48:17.241390    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5\": container with ID starting with 58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5 not found: ID does not exist" containerID="58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5"
Oct 30 23:48:17.241475 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.241417    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5"} err="failed to get container status \"58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5\": rpc error: code = NotFound desc = could not find container \"58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5\": container with ID starting with 58a77413e36c4a8c747afb41dfa945a49cc7242c3e92d6cf1a490a4eff81a8c5 not found: ID does not exist"
Oct 30 23:48:17.387178 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.387135    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"]
Oct 30 23:48:17.433460 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:17.433420    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fpbjh"]
Oct 30 23:48:18.825405 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:48:18.825364    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" path="/var/lib/kubelet/pods/e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93/volumes"
Oct 30 23:48:38.318523 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:48:38.318478    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:49:08.315777 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:49:08.315730    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:49:38.314532 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:49:38.314493    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:50:05.472039 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.472001    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"]
Oct 30 23:50:05.472515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.472296    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" containerName="maas-api"
Oct 30 23:50:05.472515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.472308    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" containerName="maas-api"
Oct 30 23:50:05.472515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.472371    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e8e1c9-1b6d-40b0-9f18-d2dff9795d93" containerName="maas-api"
Oct 30 23:50:05.475214 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.475193    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.571310 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.571271    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 23:50:05.571310 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.571304    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 23:50:05.571525 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.571345    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 23:50:05.571525 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.571384    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hg7dl\""
Oct 30 23:50:05.572432 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.572412    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"]
Oct 30 23:50:05.580803 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.580780    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7qx\" (UniqueName: \"kubernetes.io/projected/d6c3372c-3895-4d31-a94a-b7512c346169-kube-api-access-fx7qx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.580872 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.580812    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3372c-3895-4d31-a94a-b7512c346169-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.580872 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.580829    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.580975 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.580958    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.581014 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.580987    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.581053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.581042    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.681796 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.681751    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7qx\" (UniqueName: \"kubernetes.io/projected/d6c3372c-3895-4d31-a94a-b7512c346169-kube-api-access-fx7qx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.681796 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.681803    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3372c-3895-4d31-a94a-b7512c346169-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682057 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.681826    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682057 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.681869    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682057 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.681897    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682057 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.681923    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682409 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.682384    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682490 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.682404    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.682490 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.682466    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.684124 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.684102    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3372c-3895-4d31-a94a-b7512c346169-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.684388 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.684367    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3372c-3895-4d31-a94a-b7512c346169-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.740428 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.740356    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7qx\" (UniqueName: \"kubernetes.io/projected/d6c3372c-3895-4d31-a94a-b7512c346169-kube-api-access-fx7qx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg\" (UID: \"d6c3372c-3895-4d31-a94a-b7512c346169\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:05.786652 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:05.786593    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:06.148456 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:06.148425    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"]
Oct 30 23:50:06.151539 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:06.151508    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c3372c_3895_4d31_a94a_b7512c346169.slice/crio-843107a0d96c4e1441d5908cbffcccdd7d981a551938625956863dcfde5daa26 WatchSource:0}: Error finding container 843107a0d96c4e1441d5908cbffcccdd7d981a551938625956863dcfde5daa26: Status 404 returned error can't find the container with id 843107a0d96c4e1441d5908cbffcccdd7d981a551938625956863dcfde5daa26
Oct 30 23:50:06.620224 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:06.620187    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg" event={"ID":"d6c3372c-3895-4d31-a94a-b7512c346169","Type":"ContainerStarted","Data":"843107a0d96c4e1441d5908cbffcccdd7d981a551938625956863dcfde5daa26"}
Oct 30 23:50:08.315961 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:50:08.315929    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:50:08.628325 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:08.628293    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg" event={"ID":"d6c3372c-3895-4d31-a94a-b7512c346169","Type":"ContainerStarted","Data":"9a7987ef0b1df84e866231d8d7c8cf889f0bb8070b842f5d8dec334e3907b8ca"}
Oct 30 23:50:08.740785 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:08.740734    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg" podStartSLOduration=1.591271012 podStartE2EDuration="3.740718327s" totalImagesPullingTime="2.149447315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:08.739751831 +0000 UTC m=+514.030300150" watchObservedRunningTime="2026-10-30 23:50:08.740718327 +0000 UTC m=+514.031266645"
Oct 30 23:50:08.787432 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:08.787395    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:09.527678 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.527623    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"]
Oct 30 23:50:09.530905 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.530887    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.614597 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.614558    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.614597 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.614598    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/253ddce0-a6b3-41bd-91e4-b27b310d2484-kube-api-access-sb778\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.614877 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.614618    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.614877 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.614711    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/253ddce0-a6b3-41bd-91e4-b27b310d2484-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.614977 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.614871    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.614977 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.614912    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.626720 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.626686    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 23:50:09.627950 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.627927    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"]
Oct 30 23:50:09.715961 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.715918    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.715961 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.715960    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/253ddce0-a6b3-41bd-91e4-b27b310d2484-kube-api-access-sb778\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716223 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.715978    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716223 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.716005    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/253ddce0-a6b3-41bd-91e4-b27b310d2484-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716223 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.716102    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716223 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.716132    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716434 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.716408    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716495 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.716469    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.716543 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.716514    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.718228 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.718203    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/253ddce0-a6b3-41bd-91e4-b27b310d2484-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.718538 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.718522    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/253ddce0-a6b3-41bd-91e4-b27b310d2484-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.774269 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.774231    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/253ddce0-a6b3-41bd-91e4-b27b310d2484-kube-api-access-sb778\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-72tdg\" (UID: \"253ddce0-a6b3-41bd-91e4-b27b310d2484\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:09.842596 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:09.842560    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:10.207064 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:10.207036    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"]
Oct 30 23:50:10.209334 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:10.209309    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253ddce0_a6b3_41bd_91e4_b27b310d2484.slice/crio-d58a39c615073b1f5a9e4083b42c338ebc476c74527e9697624308956e484519 WatchSource:0}: Error finding container d58a39c615073b1f5a9e4083b42c338ebc476c74527e9697624308956e484519: Status 404 returned error can't find the container with id d58a39c615073b1f5a9e4083b42c338ebc476c74527e9697624308956e484519
Oct 30 23:50:10.637711 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:10.637668    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg" event={"ID":"253ddce0-a6b3-41bd-91e4-b27b310d2484","Type":"ContainerStarted","Data":"d58a39c615073b1f5a9e4083b42c338ebc476c74527e9697624308956e484519"}
Oct 30 23:50:11.643828 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:11.643789    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg" event={"ID":"253ddce0-a6b3-41bd-91e4-b27b310d2484","Type":"ContainerStarted","Data":"a7194084cc38e3a0831ac4dc4b64c75bed65697af5f38baba9b6b54e92a98288"}
Oct 30 23:50:11.755363 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:11.755304    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg" podStartSLOduration=2.157564533 podStartE2EDuration="2.755287601s" totalImagesPullingTime="597.723068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:11.753826098 +0000 UTC m=+517.044374418" watchObservedRunningTime="2026-10-30 23:50:11.755287601 +0000 UTC m=+517.045835910"
Oct 30 23:50:11.843148 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:11.843113    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:13.720260 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.720225    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"]
Oct 30 23:50:13.723544 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.723528    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.819213 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.819170    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 23:50:13.820469 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.820445    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"]
Oct 30 23:50:13.852536 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.852494    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.852724 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.852540    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.852724 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.852619    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpbk\" (UniqueName: \"kubernetes.io/projected/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-kube-api-access-kwpbk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.852808 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.852755    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.852808 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.852798    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.852883 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.852819    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.953568 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.953515    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.953568 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.953573    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.953874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.953627    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.953874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.953675    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.953874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.953704    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpbk\" (UniqueName: \"kubernetes.io/projected/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-kube-api-access-kwpbk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.953874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.953724    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.954075 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.954040    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.954126 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.954076    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.954159 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.954120    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.956064 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.956042    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:13.956343 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:13.956322    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:14.011464 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:14.011390    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpbk\" (UniqueName: \"kubernetes.io/projected/a3e63cfe-0fd2-4690-b98d-8e106e6bf421-kube-api-access-kwpbk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv\" (UID: \"a3e63cfe-0fd2-4690-b98d-8e106e6bf421\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:14.036261 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:14.036227    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:14.398284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:14.398142    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"]
Oct 30 23:50:14.400759 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:14.400734    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e63cfe_0fd2_4690_b98d_8e106e6bf421.slice/crio-43eb5b0ca1daad7d52e0d310a776ade8f639e42260bc4f3c77c6523f1287f428 WatchSource:0}: Error finding container 43eb5b0ca1daad7d52e0d310a776ade8f639e42260bc4f3c77c6523f1287f428: Status 404 returned error can't find the container with id 43eb5b0ca1daad7d52e0d310a776ade8f639e42260bc4f3c77c6523f1287f428
Oct 30 23:50:14.659727 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:14.659641    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv" event={"ID":"a3e63cfe-0fd2-4690-b98d-8e106e6bf421","Type":"ContainerStarted","Data":"43eb5b0ca1daad7d52e0d310a776ade8f639e42260bc4f3c77c6523f1287f428"}
Oct 30 23:50:15.665497 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:15.665462    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv" event={"ID":"a3e63cfe-0fd2-4690-b98d-8e106e6bf421","Type":"ContainerStarted","Data":"98da78fba914592b6bcb9f2fd1f3cefc61d7e19ba502196b983c5fb5edae000e"}
Oct 30 23:50:15.778785 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:15.778727    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv" podStartSLOduration=2.175720989 podStartE2EDuration="2.778709944s" totalImagesPullingTime="602.988955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:15.777548776 +0000 UTC m=+521.068097093" watchObservedRunningTime="2026-10-30 23:50:15.778709944 +0000 UTC m=+521.069258261"
Oct 30 23:50:15.787814 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:15.787780    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:15.800221 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:15.800188    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:16.036392 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:16.036308    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:16.680648 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:16.680610    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg"
Oct 30 23:50:18.638534 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.638501    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-44s69"]
Oct 30 23:50:18.641617 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.641596    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.691111 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.691083    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 23:50:18.738775 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.738738    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-44s69"]
Oct 30 23:50:18.797278 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.797240    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.797278 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.797280    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.797507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.797303    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnq2g\" (UniqueName: \"kubernetes.io/projected/ff7fc568-6cf9-48da-960b-860b36f5bc3f-kube-api-access-mnq2g\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.797507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.797380    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-home\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.797507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.797436    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.797507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.797478    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7fc568-6cf9-48da-960b-860b36f5bc3f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.898601 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.898520    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7fc568-6cf9-48da-960b-860b36f5bc3f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.898601 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.898573    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.898601 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.898592    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.898899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.898615    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnq2g\" (UniqueName: \"kubernetes.io/projected/ff7fc568-6cf9-48da-960b-860b36f5bc3f-kube-api-access-mnq2g\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.898899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.898660    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-home\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.898899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.898694    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.899095 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.899071    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.899165 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.899106    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-home\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.899165 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.899121    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.900877 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.900855    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7fc568-6cf9-48da-960b-860b36f5bc3f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.901432 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.901408    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7fc568-6cf9-48da-960b-860b36f5bc3f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:18.956109 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:18.956070    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnq2g\" (UniqueName: \"kubernetes.io/projected/ff7fc568-6cf9-48da-960b-860b36f5bc3f-kube-api-access-mnq2g\") pod \"echo-simulator-kserve-64b6f8f96c-44s69\" (UID: \"ff7fc568-6cf9-48da-960b-860b36f5bc3f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:19.253306 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:19.253202    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:19.615606 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:19.615574    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-44s69"]
Oct 30 23:50:19.618764 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:19.618736    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7fc568_6cf9_48da_960b_860b36f5bc3f.slice/crio-c5070fea6dd031c215b19efff1a8cf65752f55ba06c3e13856fe18f2c7251689 WatchSource:0}: Error finding container c5070fea6dd031c215b19efff1a8cf65752f55ba06c3e13856fe18f2c7251689: Status 404 returned error can't find the container with id c5070fea6dd031c215b19efff1a8cf65752f55ba06c3e13856fe18f2c7251689
Oct 30 23:50:19.683499 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:19.683463    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69" event={"ID":"ff7fc568-6cf9-48da-960b-860b36f5bc3f","Type":"ContainerStarted","Data":"c5070fea6dd031c215b19efff1a8cf65752f55ba06c3e13856fe18f2c7251689"}
Oct 30 23:50:19.842904 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:19.842873    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:19.854834 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:19.854809    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:20.699687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:20.699652    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-72tdg"
Oct 30 23:50:22.697243 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:22.697207    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69" event={"ID":"ff7fc568-6cf9-48da-960b-860b36f5bc3f","Type":"ContainerStarted","Data":"9c928ae9a693f15817b5a74a01e73a66b13aa98ca566186ecbae1848605ebc17"}
Oct 30 23:50:22.812382 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:22.812324    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69" podStartSLOduration=2.511733742 podStartE2EDuration="4.812304032s" totalImagesPullingTime="2.30057029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:22.810904266 +0000 UTC m=+528.101452585" watchObservedRunningTime="2026-10-30 23:50:22.812304032 +0000 UTC m=+528.102852350"
Oct 30 23:50:23.253680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:23.253626    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:24.036768 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:24.036720    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:24.048995 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:24.048968    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:24.715803 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:24.715772    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv"
Oct 30 23:50:25.078238 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.078162    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"]
Oct 30 23:50:25.081402 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.081386    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.130950 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.130925    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 23:50:25.138804 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.138772    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"]
Oct 30 23:50:25.255494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.255457    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae846e-f30e-48a7-93c0-04fae50b694c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.255494 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.255503    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.255782 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.255606    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.255782 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.255623    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.255782 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.255680    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.255782 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.255731    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5vz\" (UniqueName: \"kubernetes.io/projected/beae846e-f30e-48a7-93c0-04fae50b694c-kube-api-access-rz5vz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356457 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356425    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae846e-f30e-48a7-93c0-04fae50b694c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356588 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356464    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356588 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356489    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356588 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356508    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356588 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356556    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356588 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356580    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5vz\" (UniqueName: \"kubernetes.io/projected/beae846e-f30e-48a7-93c0-04fae50b694c-kube-api-access-rz5vz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356904 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356881    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.356972 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.356955    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.357098 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.357078    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.358749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.358728    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae846e-f30e-48a7-93c0-04fae50b694c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.358903 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.358887    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae846e-f30e-48a7-93c0-04fae50b694c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.414070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.414039    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5vz\" (UniqueName: \"kubernetes.io/projected/beae846e-f30e-48a7-93c0-04fae50b694c-kube-api-access-rz5vz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w\" (UID: \"beae846e-f30e-48a7-93c0-04fae50b694c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:25.703699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:25.703604    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:26.071456 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:26.071379    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"]
Oct 30 23:50:26.074359 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:26.074334    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeae846e_f30e_48a7_93c0_04fae50b694c.slice/crio-b3af9038a6c44cecc3f13a7115bd2d8bfd588c3ec15fd747dc754fb6a44095b9 WatchSource:0}: Error finding container b3af9038a6c44cecc3f13a7115bd2d8bfd588c3ec15fd747dc754fb6a44095b9: Status 404 returned error can't find the container with id b3af9038a6c44cecc3f13a7115bd2d8bfd588c3ec15fd747dc754fb6a44095b9
Oct 30 23:50:26.714535 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:26.714506    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" event={"ID":"beae846e-f30e-48a7-93c0-04fae50b694c","Type":"ContainerStarted","Data":"b3af9038a6c44cecc3f13a7115bd2d8bfd588c3ec15fd747dc754fb6a44095b9"}
Oct 30 23:50:27.720316 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:27.720281    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" event={"ID":"beae846e-f30e-48a7-93c0-04fae50b694c","Type":"ContainerStarted","Data":"2e3339562b6f4162203590492b03c531608ef4c33694a782e982bbc7ca4ba23e"}
Oct 30 23:50:27.831483 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:27.831428    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podStartSLOduration=2.243992988 podStartE2EDuration="2.831413677s" totalImagesPullingTime="587.420689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:27.830499001 +0000 UTC m=+533.121047319" watchObservedRunningTime="2026-10-30 23:50:27.831413677 +0000 UTC m=+533.121961995"
Oct 30 23:50:28.698769 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.698695    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"]
Oct 30 23:50:28.702348 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.702330    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.703799 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.703777    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:28.789711 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.789672    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.790196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.789810    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.790196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.789890    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28a2c98-8e52-474d-94c8-5441acdb2954-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.790196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.789978    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jj8\" (UniqueName: \"kubernetes.io/projected/c28a2c98-8e52-474d-94c8-5441acdb2954-kube-api-access-h8jj8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.790196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.790069    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.790196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.790113    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.798283 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.798259    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 23:50:28.799519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.799501    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"]
Oct 30 23:50:28.891036 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.890994    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891036 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891038    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891077    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891118    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891157    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28a2c98-8e52-474d-94c8-5441acdb2954-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891187    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jj8\" (UniqueName: \"kubernetes.io/projected/c28a2c98-8e52-474d-94c8-5441acdb2954-kube-api-access-h8jj8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891522 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891494    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891522 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891513    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.891718 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.891660    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.893457 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.893434    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28a2c98-8e52-474d-94c8-5441acdb2954-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.893650 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.893616    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28a2c98-8e52-474d-94c8-5441acdb2954-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:28.949397 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:28.949324    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jj8\" (UniqueName: \"kubernetes.io/projected/c28a2c98-8e52-474d-94c8-5441acdb2954-kube-api-access-h8jj8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr\" (UID: \"c28a2c98-8e52-474d-94c8-5441acdb2954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:29.014194 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.014158    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:29.254174 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.254074    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:29.266280 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.266250    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:29.381974 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.381838    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"]
Oct 30 23:50:29.384680 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:29.384656    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28a2c98_8e52_474d_94c8_5441acdb2954.slice/crio-d875df4564f42cd3aea00909b613f334a6a0055d1ad2a35d4d284791d072c29b WatchSource:0}: Error finding container d875df4564f42cd3aea00909b613f334a6a0055d1ad2a35d4d284791d072c29b: Status 404 returned error can't find the container with id d875df4564f42cd3aea00909b613f334a6a0055d1ad2a35d4d284791d072c29b
Oct 30 23:50:29.635960 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.635925    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"]
Oct 30 23:50:29.640271 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.640246    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.689580 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.689537    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:50:29.689580 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.689542    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b8qjx\""
Oct 30 23:50:29.697402 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697368    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697558 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697417    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697558 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697528    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697563    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697593    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697616    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njrrp\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697842 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697715    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697842 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697737    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.697842 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.697824    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.698095 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.698070    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"]
Oct 30 23:50:29.729019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.728984    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr" event={"ID":"c28a2c98-8e52-474d-94c8-5441acdb2954","Type":"ContainerStarted","Data":"d875df4564f42cd3aea00909b613f334a6a0055d1ad2a35d4d284791d072c29b"}
Oct 30 23:50:29.740057 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.740033    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-44s69"
Oct 30 23:50:29.798385 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798351    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.798831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798407    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.798831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798691    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.798831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798742    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.798831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798777    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.798831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798808    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798841    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798868    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njrrp\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798897    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.798924    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.799053    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799345 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.799171    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799345 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.799217    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.799447 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.799401    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.801066 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.801044    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.801274 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.801253    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.856670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.856614    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.856813 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.856746    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njrrp\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp\") pod \"maas-default-gateway-openshift-default-8f6f87f59-hkgx7\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:29.952751 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:29.952672    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:30.367098 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:30.367057    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"]
Oct 30 23:50:30.369849 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:30.369817    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56d2a7a_f98b_4fed_b3fa_291df3265d4c.slice/crio-29bbe6f5e19c763f3bc1c573c44a1f6cbbf04ab694b8178187ef92e2f85406b7 WatchSource:0}: Error finding container 29bbe6f5e19c763f3bc1c573c44a1f6cbbf04ab694b8178187ef92e2f85406b7: Status 404 returned error can't find the container with id 29bbe6f5e19c763f3bc1c573c44a1f6cbbf04ab694b8178187ef92e2f85406b7
Oct 30 23:50:30.735200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:30.735115    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr" event={"ID":"c28a2c98-8e52-474d-94c8-5441acdb2954","Type":"ContainerStarted","Data":"3c28cd82c7ef4ef5c3007498985b1e75f3c97d6b574357b077be0150b87df53f"}
Oct 30 23:50:30.736387 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:30.736362    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" event={"ID":"b56d2a7a-f98b-4fed-b3fa-291df3265d4c","Type":"ContainerStarted","Data":"29bbe6f5e19c763f3bc1c573c44a1f6cbbf04ab694b8178187ef92e2f85406b7"}
Oct 30 23:50:30.847437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:30.847383    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr" podStartSLOduration=2.200542698 podStartE2EDuration="2.847367028s" totalImagesPullingTime="646.82433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:30.845996806 +0000 UTC m=+536.136545124" watchObservedRunningTime="2026-10-30 23:50:30.847367028 +0000 UTC m=+536.137915346"
Oct 30 23:50:31.015198 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:31.015118    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:34.858385 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:34.858356    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:50:34.858681 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:34.858420    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:50:34.858681 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:34.858449    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:50:35.704553 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.704522    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:35.722123 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.722096    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:35.760294 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.760257    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" event={"ID":"b56d2a7a-f98b-4fed-b3fa-291df3265d4c","Type":"ContainerStarted","Data":"29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06"}
Oct 30 23:50:35.771115 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.771087    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 30 23:50:35.953903 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.953858    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:35.955742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.955673    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 23:50:35.955742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:35.955728    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 23:50:36.077348 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.077291    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podStartSLOduration=2.591815176 podStartE2EDuration="7.077275202s" totalImagesPullingTime="4.485460026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:36.075676245 +0000 UTC m=+541.366224563" watchObservedRunningTime="2026-10-30 23:50:36.077275202 +0000 UTC m=+541.367823519"
Oct 30 23:50:36.165653 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.165600    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"]
Oct 30 23:50:36.169007 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.168992    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.218657 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.218563    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 23:50:36.226646 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.226605    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"]
Oct 30 23:50:36.258583 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.258536    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.258760 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.258696    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.258760 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.258730    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.258876 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.258813    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.258927 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.258900    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8hh\" (UniqueName: \"kubernetes.io/projected/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-kube-api-access-sc8hh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.258979 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.258959    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.359813 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.359776    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.359986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.359867    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.359986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.359896    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.359986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.359960    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.360123 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.359999    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8hh\" (UniqueName: \"kubernetes.io/projected/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-kube-api-access-sc8hh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.360123 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.360048    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.360275 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.360255    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.360358 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.360329    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.360473 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.360393    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.362162 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.362140    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.362461 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.362441    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.418194 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.418162    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8hh\" (UniqueName: \"kubernetes.io/projected/c4d60297-7ba5-4d8a-8c01-0a75a61b0e48-kube-api-access-sc8hh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-274g7\" (UID: \"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.480933 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.480846    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:36.864774 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.864662    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"]
Oct 30 23:50:36.867454 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:36.867422    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d60297_7ba5_4d8a_8c01_0a75a61b0e48.slice/crio-4496749f350279426c810c1ec79c3ab8034eacad4e695a45fa02abfca61a5ee4 WatchSource:0}: Error finding container 4496749f350279426c810c1ec79c3ab8034eacad4e695a45fa02abfca61a5ee4: Status 404 returned error can't find the container with id 4496749f350279426c810c1ec79c3ab8034eacad4e695a45fa02abfca61a5ee4
Oct 30 23:50:36.954128 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.954081    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 23:50:36.954529 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:36.954150    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 23:50:37.770832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:37.770741    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7" event={"ID":"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48","Type":"ContainerStarted","Data":"db894df3bea32c7e8e3dc14636bb69be3ee0e595299d1fbbf6663497112f8c69"}
Oct 30 23:50:37.770832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:37.770801    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7" event={"ID":"c4d60297-7ba5-4d8a-8c01-0a75a61b0e48","Type":"ContainerStarted","Data":"4496749f350279426c810c1ec79c3ab8034eacad4e695a45fa02abfca61a5ee4"}
Oct 30 23:50:37.885285 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:37.885234    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7" podStartSLOduration=1.261074089 podStartE2EDuration="1.885218667s" totalImagesPullingTime="624.144578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:37.883848825 +0000 UTC m=+543.174397143" watchObservedRunningTime="2026-10-30 23:50:37.885218667 +0000 UTC m=+543.175766984"
Oct 30 23:50:37.954074 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:37.954035    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 23:50:37.954274 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:37.954113    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 23:50:38.315216 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:50:38.315176    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:50:38.481714 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:38.481671    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:38.954200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:38.954159    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 23:50:38.954386 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:38.954225    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 23:50:39.015042 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:39.015011    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:39.027213 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:39.027183    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:39.794670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:39.794622    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr"
Oct 30 23:50:39.953052 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:39.953018    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 23:50:39.953309 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:39.953283    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 23:50:40.953958 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:40.953916    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 23:50:40.954335 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:40.953990    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 23:50:42.954067 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:42.954022    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:50:42.954496 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:42.954098    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:50:43.996829 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:43.996781    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:50:43.997340 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:43.996857    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:50:44.050008 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:44.049977    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:44.050393 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:44.050370    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:44.163711 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:44.163681    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 30 23:50:46.481174 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:46.481135    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:46.493554 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:46.493529    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:46.819519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:46.819439    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-274g7"
Oct 30 23:50:54.126786 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.126741    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-7b62j"]
Oct 30 23:50:54.134650 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.134606    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.185928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.185895    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-7b62j"]
Oct 30 23:50:54.217501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.217457    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljzj\" (UniqueName: \"kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj\") pod \"ai-gateway-controller-7d6c696b49-7b62j\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.217704 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.217507    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-7b62j\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.230871 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.230845    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 23:50:54.230951 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.230873    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 23:50:54.234536 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.234506    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:50:54.234680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.234551    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:50:54.234680 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.234676    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-76z49\""
Oct 30 23:50:54.319166 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.319035    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljzj\" (UniqueName: \"kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj\") pod \"ai-gateway-controller-7d6c696b49-7b62j\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.319166 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.319088    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-7b62j\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.327591 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.321531    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-7b62j\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.377331 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.377254    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljzj\" (UniqueName: \"kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj\") pod \"ai-gateway-controller-7d6c696b49-7b62j\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.449193 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.449158    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:54.817765 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.817565    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-7b62j"]
Oct 30 23:50:54.820805 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:50:54.820460    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315437b8_c38f_4a8a_ab29_a386c1337ebd.slice/crio-05a84aa8ba46fd5ffde37129d2b6a3caab1ff1d720de62eb57b84645f7995d30 WatchSource:0}: Error finding container 05a84aa8ba46fd5ffde37129d2b6a3caab1ff1d720de62eb57b84645f7995d30: Status 404 returned error can't find the container with id 05a84aa8ba46fd5ffde37129d2b6a3caab1ff1d720de62eb57b84645f7995d30
Oct 30 23:50:54.845935 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:54.845903    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j" event={"ID":"315437b8-c38f-4a8a-ab29-a386c1337ebd","Type":"ContainerStarted","Data":"05a84aa8ba46fd5ffde37129d2b6a3caab1ff1d720de62eb57b84645f7995d30"}
Oct 30 23:50:57.860151 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:57.860118    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j" event={"ID":"315437b8-c38f-4a8a-ab29-a386c1337ebd","Type":"ContainerStarted","Data":"74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600"}
Oct 30 23:50:57.860598 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:57.860240    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:50:57.969581 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:57.969509    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j" podStartSLOduration=1.721302916 podStartE2EDuration="3.969491651s" totalImagesPullingTime="2.248188735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:57.969251916 +0000 UTC m=+563.259800241" watchObservedRunningTime="2026-10-30 23:50:57.969491651 +0000 UTC m=+563.260039970"
Oct 30 23:50:59.884251 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.884215    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"]
Oct 30 23:50:59.887612 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.887596    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975695 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975649    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975851 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975763    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975851 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975807    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78csc\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975851 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975827    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975851 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975847    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975875    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975906    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975937    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.975986 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.975966    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:50:59.984744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:50:59.984711    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"]
Oct 30 23:51:00.076823 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.076783    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78csc\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.076829    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.076859    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.076894    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.076933    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.076973    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077013    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077337 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077044    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077337 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077094    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077337 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077311    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077510 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077363    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077510 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077424    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077593 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077511    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.077887 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.077861    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.079354 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.079319    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.079716 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.079694    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.134453 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.134378    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78csc\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.134453 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.134386    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-bd4hv\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.199740 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.199705    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:00.617293 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.617266    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"]
Oct 30 23:51:00.619250 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:51:00.619226    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bdbd9e_2149_4c46_b48c_5c1ebea3aafd.slice/crio-1e84eeabfca7c6c82be5955c6d50f39f91adff24f626d1404ef110a02810c3d5 WatchSource:0}: Error finding container 1e84eeabfca7c6c82be5955c6d50f39f91adff24f626d1404ef110a02810c3d5: Status 404 returned error can't find the container with id 1e84eeabfca7c6c82be5955c6d50f39f91adff24f626d1404ef110a02810c3d5
Oct 30 23:51:00.621335 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.621299    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:00.621452 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.621365    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:00.621452 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.621398    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:00.873215 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.873133    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" event={"ID":"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd","Type":"ContainerStarted","Data":"27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4"}
Oct 30 23:51:00.873215 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.873183    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" event={"ID":"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd","Type":"ContainerStarted","Data":"1e84eeabfca7c6c82be5955c6d50f39f91adff24f626d1404ef110a02810c3d5"}
Oct 30 23:51:00.994243 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:00.994183    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podStartSLOduration=1.994163231 podStartE2EDuration="1.994163231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:00.992016693 +0000 UTC m=+566.282565013" watchObservedRunningTime="2026-10-30 23:51:00.994163231 +0000 UTC m=+566.284711550"
Oct 30 23:51:01.200245 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:01.200151    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:01.201689 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:01.201664    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:01.201813 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:01.201722    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:02.201196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:02.201152    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:02.201587 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:02.201214    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:03.201073 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:03.201034    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:03.201258 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:03.201108    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:04.201032 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:04.200995    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:04.201212 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:04.201051    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:05.200527 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:05.200495    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:05.200931 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:05.200550    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:06.200827 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:06.200790    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:06.201236 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:06.200862    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:07.200767 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:07.200726    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-bd4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 23:51:07.200946 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:07.200787    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 23:51:08.322743 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:51:08.322704    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:51:08.768720 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:08.768686    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:08.769228 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:08.769196    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:08.770244 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:08.770226    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 30 23:51:08.866091 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:08.866058    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:51:09.417613 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.417579    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"]
Oct 30 23:51:09.426193 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.426170    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:09.482661 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.480449    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"]
Oct 30 23:51:09.589242 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.589200    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkgf\" (UniqueName: \"kubernetes.io/projected/eba33c69-bc96-416a-80a6-199992fccebd-kube-api-access-txkgf\") pod \"ai-gateway-controller-86cfd56d94-c7bc6\" (UID: \"eba33c69-bc96-416a-80a6-199992fccebd\") " pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:09.589422 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.589331    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba33c69-bc96-416a-80a6-199992fccebd-metrics-certs\") pod \"ai-gateway-controller-86cfd56d94-c7bc6\" (UID: \"eba33c69-bc96-416a-80a6-199992fccebd\") " pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:09.690468 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.690380    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkgf\" (UniqueName: \"kubernetes.io/projected/eba33c69-bc96-416a-80a6-199992fccebd-kube-api-access-txkgf\") pod \"ai-gateway-controller-86cfd56d94-c7bc6\" (UID: \"eba33c69-bc96-416a-80a6-199992fccebd\") " pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:09.690656 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.690473    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba33c69-bc96-416a-80a6-199992fccebd-metrics-certs\") pod \"ai-gateway-controller-86cfd56d94-c7bc6\" (UID: \"eba33c69-bc96-416a-80a6-199992fccebd\") " pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:09.693146 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.693108    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba33c69-bc96-416a-80a6-199992fccebd-metrics-certs\") pod \"ai-gateway-controller-86cfd56d94-c7bc6\" (UID: \"eba33c69-bc96-416a-80a6-199992fccebd\") " pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:09.748793 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:09.748763    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkgf\" (UniqueName: \"kubernetes.io/projected/eba33c69-bc96-416a-80a6-199992fccebd-kube-api-access-txkgf\") pod \"ai-gateway-controller-86cfd56d94-c7bc6\" (UID: \"eba33c69-bc96-416a-80a6-199992fccebd\") " pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:10.039594 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:10.039497    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:10.415530 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:10.415493    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"]
Oct 30 23:51:10.418507 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:51:10.418478    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba33c69_bc96_416a_80a6_199992fccebd.slice/crio-3d2ffe293b194a0245fd06fda5a2e681b395f3ece0400061a61bac6a462bf67c WatchSource:0}: Error finding container 3d2ffe293b194a0245fd06fda5a2e681b395f3ece0400061a61bac6a462bf67c: Status 404 returned error can't find the container with id 3d2ffe293b194a0245fd06fda5a2e681b395f3ece0400061a61bac6a462bf67c
Oct 30 23:51:10.915852 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:10.915816    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6" event={"ID":"eba33c69-bc96-416a-80a6-199992fccebd","Type":"ContainerStarted","Data":"3d2ffe293b194a0245fd06fda5a2e681b395f3ece0400061a61bac6a462bf67c"}
Oct 30 23:51:11.921841 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:11.921807    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6" event={"ID":"eba33c69-bc96-416a-80a6-199992fccebd","Type":"ContainerStarted","Data":"26eaffca78c673c075dd07968a77c41cba1650f844aa3d94815a7580f8c1a1f9"}
Oct 30 23:51:11.922221 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:11.921890    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:12.032046 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:12.031991    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6" podStartSLOduration=2.041555409 podStartE2EDuration="3.031974564s" totalImagesPullingTime="990.419155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:12.031383875 +0000 UTC m=+577.321932189" watchObservedRunningTime="2026-10-30 23:51:12.031974564 +0000 UTC m=+577.322522882"
Oct 30 23:51:22.928781 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:22.928700    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86cfd56d94-c7bc6"
Oct 30 23:51:23.091557 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.091522    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-7b62j"]
Oct 30 23:51:23.091800 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.091777    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j" podUID="315437b8-c38f-4a8a-ab29-a386c1337ebd" containerName="manager" containerID="cri-o://74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600" gracePeriod=30
Oct 30 23:51:23.337312 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.337288    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:51:23.411654 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.411609    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj\" (UniqueName: \"kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj\") pod \"315437b8-c38f-4a8a-ab29-a386c1337ebd\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") "
Oct 30 23:51:23.411830 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.411719    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs\") pod \"315437b8-c38f-4a8a-ab29-a386c1337ebd\" (UID: \"315437b8-c38f-4a8a-ab29-a386c1337ebd\") "
Oct 30 23:51:23.413949 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.413912    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs" pod "315437b8-c38f-4a8a-ab29-a386c1337ebd" (UID: "315437b8-c38f-4a8a-ab29-a386c1337ebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:51:23.413949 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.413935    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj" pod "315437b8-c38f-4a8a-ab29-a386c1337ebd" (UID: "315437b8-c38f-4a8a-ab29-a386c1337ebd"). InnerVolumeSpecName "kube-api-access-nljzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:51:23.512844 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.512760    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315437b8-c38f-4a8a-ab29-a386c1337ebd-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:51:23.512844 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.512792    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljzj\" (UniqueName: \"kubernetes.io/projected/315437b8-c38f-4a8a-ab29-a386c1337ebd-kube-api-access-nljzj\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:51:23.971728 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.971695    4466 generic.go:356] "Generic (PLEG): container finished" podID="315437b8-c38f-4a8a-ab29-a386c1337ebd" containerID="74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600" exitCode=0
Oct 30 23:51:23.972210 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.971744    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j"
Oct 30 23:51:23.972210 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.971760    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j" event={"ID":"315437b8-c38f-4a8a-ab29-a386c1337ebd","Type":"ContainerDied","Data":"74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600"}
Oct 30 23:51:23.972210 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.971799    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-7b62j" event={"ID":"315437b8-c38f-4a8a-ab29-a386c1337ebd","Type":"ContainerDied","Data":"05a84aa8ba46fd5ffde37129d2b6a3caab1ff1d720de62eb57b84645f7995d30"}
Oct 30 23:51:23.972210 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.971820    4466 scope.go:122] "RemoveContainer" containerID="74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600"
Oct 30 23:51:23.980753 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.980733    4466 scope.go:122] "RemoveContainer" containerID="74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600"
Oct 30 23:51:23.981023 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:51:23.981001    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600\": container with ID starting with 74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600 not found: ID does not exist" containerID="74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600"
Oct 30 23:51:23.981088 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:23.981037    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600"} err="failed to get container status \"74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600\": rpc error: code = NotFound desc = could not find container \"74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600\": container with ID starting with 74602110b4433d3739b19ca3e1d16947732e0dcc21de50ca9bebdada2e903600 not found: ID does not exist"
Oct 30 23:51:24.125382 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:24.125349    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-7b62j"]
Oct 30 23:51:24.172036 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:24.172001    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-7b62j"]
Oct 30 23:51:24.825585 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:24.825551    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315437b8-c38f-4a8a-ab29-a386c1337ebd" path="/var/lib/kubelet/pods/315437b8-c38f-4a8a-ab29-a386c1337ebd/volumes"
Oct 30 23:51:38.315572 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:51:38.315533    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:51:54.295746 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.295711    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"]
Oct 30 23:51:54.296260 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.296060    4466 cpu_manager.go:414] "containerMap: removing container" podUID="315437b8-c38f-4a8a-ab29-a386c1337ebd" containerName="manager"
Oct 30 23:51:54.296260 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.296070    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315437b8-c38f-4a8a-ab29-a386c1337ebd" containerName="manager"
Oct 30 23:51:54.296260 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.296138    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="315437b8-c38f-4a8a-ab29-a386c1337ebd" containerName="manager"
Oct 30 23:51:54.299083 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.299066    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.348683 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.348653    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-34f1ce-openshift-default-dockercfg-gfqnh\""
Oct 30 23:51:54.396398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.396360    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"]
Oct 30 23:51:54.400681 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400652    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.400681 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400684    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.400898 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400702    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxhs\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.400898 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400727    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.400898 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400842    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.401043 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400912    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.401043 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400939    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.401043 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.400988    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.401043 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.401041    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.501858 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.501820    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.501858 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.501863    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502097 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.501913    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502097 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502014    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502097 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502039    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502097 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502064    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxhs\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502107    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502134    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502176    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502295    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502502 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502454    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502585 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502569    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502666 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502620    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.502788 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.502770    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.504488 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.504464    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.504587 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.504566    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.560015 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.559936    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.560174 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.560099    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxhs\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.611137 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.611100    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:54.946537 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.946504    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"]
Oct 30 23:51:54.951205 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:54.951180    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.000427 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.000399    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-27e82b-openshift-default-dockercfg-2pkcf\""
Oct 30 23:51:55.009111 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.009078    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"]
Oct 30 23:51:55.083331 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.083299    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"]
Oct 30 23:51:55.089318 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.089288    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:55.089387 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.089354    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:55.089387 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.089379    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:55.095075 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.095040    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" event={"ID":"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5","Type":"ContainerStarted","Data":"1cd6a4f826684b730e46d6fef3306c5492ca56ce453d83a0a77e79f2efd6b47b"}
Oct 30 23:51:55.109036 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109008    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109160 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109075    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109252 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109207    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109303 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109267    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc9p\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109373 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109355    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109413 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109393    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109477 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109461    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109492    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.109565 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.109531    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211007    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211069    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211103 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211087    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc9p\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211113    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211136    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211159    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211383 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211175    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211528    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211537    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211602    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211617    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.211960 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.211935    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.212092 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.212069    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.212277 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.212255    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.213353 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.213328    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.213752 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.213732    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.276384 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.276352    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc9p\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.278312 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.278294    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.564664 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.564539    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:55.607973 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.607937    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"]
Oct 30 23:51:55.620900 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.620868    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.671137 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.670493    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-46b586-openshift-default-dockercfg-5bltr\""
Oct 30 23:51:55.671304 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.671194    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"]
Oct 30 23:51:55.718758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.718723    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.718924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.718771    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.718924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.718788    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.718924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.718893    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcng\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.719021 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.718949    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.719021 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.718982    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.719021 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.719003    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.719114 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.719057    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.719146 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.719139    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820145 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820072    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820291 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820149    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820291 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820189    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820291 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820212    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820291 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820247    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcng\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820513 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820304    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820513 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820339    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820513 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820371    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820513 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820397    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820760 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820506    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820760 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820744    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820875 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820854    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.820936 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.820901    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.821158 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.821138    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.822742 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.822719    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.822846 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.822741    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.890071 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.890034    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.890212 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.890135    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcng\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.936855 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.936823    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:55.987626 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.987597    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"]
Oct 30 23:51:55.995709 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.994103    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"]
Oct 30 23:51:55.995709 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.994197    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:55.997409 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:51:55.997376    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f335002_5dbf_48dc_8f24_0080043e516f.slice/crio-d50c956933124493704a55c672b24f61e45b9c166d1922f258a2a5aea711928d WatchSource:0}: Error finding container d50c956933124493704a55c672b24f61e45b9c166d1922f258a2a5aea711928d: Status 404 returned error can't find the container with id d50c956933124493704a55c672b24f61e45b9c166d1922f258a2a5aea711928d
Oct 30 23:51:55.999870 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.999832    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:55.999961 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.999917    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:55.999961 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:55.999953    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:56.053685 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.053659    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fe01fa-openshift-default-dockercfg-5mw89\""
Oct 30 23:51:56.054288 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.054266    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"]
Oct 30 23:51:56.101854 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.101809    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" event={"ID":"5f335002-5dbf-48dc-8f24-0080043e516f","Type":"ContainerStarted","Data":"edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"}
Oct 30 23:51:56.102032 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.101891    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" event={"ID":"5f335002-5dbf-48dc-8f24-0080043e516f","Type":"ContainerStarted","Data":"d50c956933124493704a55c672b24f61e45b9c166d1922f258a2a5aea711928d"}
Oct 30 23:51:56.103521 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.103490    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" event={"ID":"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5","Type":"ContainerStarted","Data":"311993b736c51d5b6e46ee8f85496f63e80df0a0dd5bee8c00208098fecf23b2"}
Oct 30 23:51:56.123416 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123380    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123586 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123476    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123586 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123523    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123701 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123608    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123701 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123646    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123809 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123793    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123878 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123856    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.123933 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.123917    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.124019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.124006    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkh4\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.224901 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.224866    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.224901 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.224906    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225152 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.224958    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225152 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225005    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkh4\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225152 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225084    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225152 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225109    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225152 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225143    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225407 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225193    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225407 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225220    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.225690 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.225664    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.226075 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.226037    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.226170 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.226069    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.226170 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.226150    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.226506 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.226483    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.227978 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.227952    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.228099 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.228084    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.231041 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.231003    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podStartSLOduration=2.230990562 podStartE2EDuration="2.230990562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:56.229555298 +0000 UTC m=+621.520103626" watchObservedRunningTime="2026-10-30 23:51:56.230990562 +0000 UTC m=+621.521538957"
Oct 30 23:51:56.283585 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.283549    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.283734 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.283596    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkh4\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.303229 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.303185    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"]
Oct 30 23:51:56.305980 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:51:56.305953    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90e56b0_a1e4_42ad_bd0f_e26a3cd96e32.slice/crio-fce92b9bde7b1e059cf0e8fc96203c6113a94668efe345ba28f5369120f6e0a0 WatchSource:0}: Error finding container fce92b9bde7b1e059cf0e8fc96203c6113a94668efe345ba28f5369120f6e0a0: Status 404 returned error can't find the container with id fce92b9bde7b1e059cf0e8fc96203c6113a94668efe345ba28f5369120f6e0a0
Oct 30 23:51:56.308800 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.308768    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:56.308905 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.308834    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:56.308905 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.308864    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:56.311796 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.311774    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:56.344899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.344846    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podStartSLOduration=2.344824515 podStartE2EDuration="2.344824515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:56.343099265 +0000 UTC m=+621.633647584" watchObservedRunningTime="2026-10-30 23:51:56.344824515 +0000 UTC m=+621.635372835"
Oct 30 23:51:56.565691 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.565654    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:56.611844 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.611748    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:56.617045 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.617016    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:56.693660 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.693537    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"]
Oct 30 23:51:56.696431 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:51:56.696401    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e83d72a_ac30_48b1_86f1_83b0f29514f7.slice/crio-e3b3b51bbb663275cd53c0e55342412c6346c3135f73cdd78013c82e7dd0792f WatchSource:0}: Error finding container e3b3b51bbb663275cd53c0e55342412c6346c3135f73cdd78013c82e7dd0792f: Status 404 returned error can't find the container with id e3b3b51bbb663275cd53c0e55342412c6346c3135f73cdd78013c82e7dd0792f
Oct 30 23:51:56.698965 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.698933    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:56.699089 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.699000    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:56.699089 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:56.699028    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:51:57.109088 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.109051    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" event={"ID":"9e83d72a-ac30-48b1-86f1-83b0f29514f7","Type":"ContainerStarted","Data":"888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"}
Oct 30 23:51:57.109270 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.109116    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" event={"ID":"9e83d72a-ac30-48b1-86f1-83b0f29514f7","Type":"ContainerStarted","Data":"e3b3b51bbb663275cd53c0e55342412c6346c3135f73cdd78013c82e7dd0792f"}
Oct 30 23:51:57.110649 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.110615    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" event={"ID":"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32","Type":"ContainerStarted","Data":"dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"}
Oct 30 23:51:57.110762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.110676    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" event={"ID":"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32","Type":"ContainerStarted","Data":"fce92b9bde7b1e059cf0e8fc96203c6113a94668efe345ba28f5369120f6e0a0"}
Oct 30 23:51:57.110970 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.110940    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:57.111776 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.111760    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:51:57.224512 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.224455    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podStartSLOduration=2.224439146 podStartE2EDuration="2.224439146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:57.222882246 +0000 UTC m=+622.513430565" watchObservedRunningTime="2026-10-30 23:51:57.224439146 +0000 UTC m=+622.514987464"
Oct 30 23:51:57.312259 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.312227    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:57.317576 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.317548    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:57.337979 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.337818    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podStartSLOduration=2.337749979 podStartE2EDuration="2.337749979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:57.336225463 +0000 UTC m=+622.626773781" watchObservedRunningTime="2026-10-30 23:51:57.337749979 +0000 UTC m=+622.628298363"
Oct 30 23:51:57.570204 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.570131    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:57.937652 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.937613    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:57.942621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:57.942595    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:58.120389 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:58.120352    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:58.120801 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:58.120775    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:51:58.120931 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:58.120824    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:58.121098 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:58.121083    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:51:58.121167 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:58.121127    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:51:58.121269 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:51:58.121257    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:52:05.045387 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.045357    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"]
Oct 30 23:52:05.050250 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.050230    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.102835 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.102801    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-27e82b-serving-cert\""
Oct 30 23:52:05.110615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.110575    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.110799 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.110761    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxd6m\" (UniqueName: \"kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.110881 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.110807    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.110954 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.110927    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.148485 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.148447    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"]
Oct 30 23:52:05.212277 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.212236    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.212536 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.212338    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxd6m\" (UniqueName: \"kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.212536 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.212379    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.212536 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.212443    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.212536 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:05.212484    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-serving-cert: secret "maas-api-e2e-worker-w2-27e82b-serving-cert" not found
Oct 30 23:52:05.212771 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:05.212549    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls podName:fd6de883-96fc-4b16-b843-fbb8932dfd74 nodeName:}" failed. No retries permitted until 2026-10-30 23:52:05.712526915 +0000 UTC m=+631.003075225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls") pod "maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" (UID: "fd6de883-96fc-4b16-b843-fbb8932dfd74") : secret "maas-api-e2e-worker-w2-27e82b-serving-cert" not found
Oct 30 23:52:05.213123 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.213104    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.214603 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.214581    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.272391 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.272359    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxd6m\" (UniqueName: \"kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.716459 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.716427    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.718890 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.718865    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\") pod \"maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:05.961819 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:05.961786    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:06.326284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:06.326237    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"]
Oct 30 23:52:06.328766 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:06.328734    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6de883_96fc_4b16_b843_fbb8932dfd74.slice/crio-9e68a2c400dbda1ce151dfab36227269d9da683129dd6f892d8554fe812077fa WatchSource:0}: Error finding container 9e68a2c400dbda1ce151dfab36227269d9da683129dd6f892d8554fe812077fa: Status 404 returned error can't find the container with id 9e68a2c400dbda1ce151dfab36227269d9da683129dd6f892d8554fe812077fa
Oct 30 23:52:06.330267 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:06.330247    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:52:07.159842 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:07.159802    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" event={"ID":"fd6de883-96fc-4b16-b843-fbb8932dfd74","Type":"ContainerStarted","Data":"9e68a2c400dbda1ce151dfab36227269d9da683129dd6f892d8554fe812077fa"}
Oct 30 23:52:08.166366 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:08.166323    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" event={"ID":"fd6de883-96fc-4b16-b843-fbb8932dfd74","Type":"ContainerStarted","Data":"b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4"}
Oct 30 23:52:08.166749 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:08.166479    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:08.279621 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:08.279562    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" podStartSLOduration=2.320451659 podStartE2EDuration="3.279544695s" totalImagesPullingTime="959.093036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:08.277565293 +0000 UTC m=+633.568113612" watchObservedRunningTime="2026-10-30 23:52:08.279544695 +0000 UTC m=+633.570093012"
Oct 30 23:52:08.316617 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:08.316584    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:52:10.043091 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.043054    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"]
Oct 30 23:52:10.046748 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.046730    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.095747 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.095716    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-34f1ce-serving-cert\""
Oct 30 23:52:10.144037 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.144001    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"]
Oct 30 23:52:10.166568 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.166537    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.166732 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.166579    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgfq\" (UniqueName: \"kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.166732 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.166676    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.166732 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.166723    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.268139 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.268108    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.268321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.268156    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgfq\" (UniqueName: \"kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.268321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.268188    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.268321 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.268238    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.268887 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.268864    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.270765 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.270740    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.270849 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.270746    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.325758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.325678    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgfq\" (UniqueName: \"kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq\") pod \"maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.359125 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.359094    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:10.726610 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:10.726586    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"]
Oct 30 23:52:10.728756 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:10.728727    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0091c1_6953_4cee_8eb7_54926c156ea5.slice/crio-fbe69d0623bbe46ea06d3d4561cbd82f6d19a616395119b96ff77bc7285bbc5a WatchSource:0}: Error finding container fbe69d0623bbe46ea06d3d4561cbd82f6d19a616395119b96ff77bc7285bbc5a: Status 404 returned error can't find the container with id fbe69d0623bbe46ea06d3d4561cbd82f6d19a616395119b96ff77bc7285bbc5a
Oct 30 23:52:11.179876 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:11.179842    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn" event={"ID":"6c0091c1-6953-4cee-8eb7-54926c156ea5","Type":"ContainerStarted","Data":"fbe69d0623bbe46ea06d3d4561cbd82f6d19a616395119b96ff77bc7285bbc5a"}
Oct 30 23:52:12.185506 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:12.185466    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn" event={"ID":"6c0091c1-6953-4cee-8eb7-54926c156ea5","Type":"ContainerStarted","Data":"cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29"}
Oct 30 23:52:12.185921 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:12.185599    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:12.310113 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:12.310049    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn" podStartSLOduration=2.389619378 podStartE2EDuration="3.310033693s" totalImagesPullingTime="920.414315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:12.308849874 +0000 UTC m=+637.599398194" watchObservedRunningTime="2026-10-30 23:52:12.310033693 +0000 UTC m=+637.600582011"
Oct 30 23:52:14.176263 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:14.176234    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 30 23:52:15.153825 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.153788    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"]
Oct 30 23:52:15.157852 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.157829    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.205859 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.205818    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"]
Oct 30 23:52:15.221936 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.221906    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285mv\" (UniqueName: \"kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.222109 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.221975    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.222109 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.222069    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.222212 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.222117    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.253802 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.253765    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5c3374-serving-cert\""
Oct 30 23:52:15.323600 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.323562    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285mv\" (UniqueName: \"kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.323787 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.323667    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.323787 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.323728    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.323914 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.323795    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.323914 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:15.323898    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-serving-cert: secret "maas-api-e2e-worker-w1-5c3374-serving-cert" not found
Oct 30 23:52:15.324025 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:15.323957    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls podName:b756e4c9-eb7f-4602-986f-1cf34e213542 nodeName:}" failed. No retries permitted until 2026-10-30 23:52:15.823938786 +0000 UTC m=+641.114487098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls") pod "maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" (UID: "b756e4c9-eb7f-4602-986f-1cf34e213542") : secret "maas-api-e2e-worker-w1-5c3374-serving-cert" not found
Oct 30 23:52:15.324366 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.324340    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.326129 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.326108    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.383896 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.383861    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-285mv\" (UniqueName: \"kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.835668 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.834282    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:15.843306 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:15.843273    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:16.070620 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:16.070580    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:16.450914 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:16.448792    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"]
Oct 30 23:52:16.454477 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:16.454444    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb756e4c9_eb7f_4602_986f_1cf34e213542.slice/crio-db2f86f7c86d2c5fd74db7b2a6a012eda38c61e2dc5adc6c0cc6fe7abba3e265 WatchSource:0}: Error finding container db2f86f7c86d2c5fd74db7b2a6a012eda38c61e2dc5adc6c0cc6fe7abba3e265: Status 404 returned error can't find the container with id db2f86f7c86d2c5fd74db7b2a6a012eda38c61e2dc5adc6c0cc6fe7abba3e265
Oct 30 23:52:17.210723 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:17.210680    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" event={"ID":"b756e4c9-eb7f-4602-986f-1cf34e213542","Type":"ContainerStarted","Data":"db2f86f7c86d2c5fd74db7b2a6a012eda38c61e2dc5adc6c0cc6fe7abba3e265"}
Oct 30 23:52:18.196211 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:18.196180    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 30 23:52:18.219047 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:18.219009    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" event={"ID":"b756e4c9-eb7f-4602-986f-1cf34e213542","Type":"ContainerStarted","Data":"babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7"}
Oct 30 23:52:18.219227 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:18.219154    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:18.421099 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:18.421035    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" podStartSLOduration=2.59589781 podStartE2EDuration="3.421014719s" totalImagesPullingTime="825.116909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:18.419568327 +0000 UTC m=+643.710116645" watchObservedRunningTime="2026-10-30 23:52:18.421014719 +0000 UTC m=+643.711563043"
Oct 30 23:52:20.038660 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.038604    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"]
Oct 30 23:52:20.042205 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.042186    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.091278 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.091248    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-35a187-serving-cert\""
Oct 30 23:52:20.141484 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.141442    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"]
Oct 30 23:52:20.202327 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.202285    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzq6\" (UniqueName: \"kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.202541 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.202340    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.202541 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.202441    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.202685 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.202566    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.303618 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.303576    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.303945 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.303920    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzq6\" (UniqueName: \"kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.304067 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.303960    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.304067 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.303984    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.304789 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.304758    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.306578 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.306554    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.306689 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.306592    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.362180 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.362101    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzq6\" (UniqueName: \"kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6\") pod \"maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:20.655398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:20.655302    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:21.040296 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:21.040270    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"]
Oct 30 23:52:21.043551 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:21.043496    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc82656a_5038_49ec_8910_82069ea2905a.slice/crio-d4423fc940f5560063e8ac99c595618df9f8e7e476d7d97489284a76c1d7ae89 WatchSource:0}: Error finding container d4423fc940f5560063e8ac99c595618df9f8e7e476d7d97489284a76c1d7ae89: Status 404 returned error can't find the container with id d4423fc940f5560063e8ac99c595618df9f8e7e476d7d97489284a76c1d7ae89
Oct 30 23:52:21.234154 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:21.234112    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r" event={"ID":"dc82656a-5038-49ec-8910-82069ea2905a","Type":"ContainerStarted","Data":"d4423fc940f5560063e8ac99c595618df9f8e7e476d7d97489284a76c1d7ae89"}
Oct 30 23:52:22.240701 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:22.240665    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r" event={"ID":"dc82656a-5038-49ec-8910-82069ea2905a","Type":"ContainerStarted","Data":"cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520"}
Oct 30 23:52:22.241134 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:22.240725    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:22.352493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:22.352439    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r" podStartSLOduration=2.278725581 podStartE2EDuration="3.352421745s" totalImagesPullingTime="1.073696164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:22.351073695 +0000 UTC m=+647.641622014" watchObservedRunningTime="2026-10-30 23:52:22.352421745 +0000 UTC m=+647.642970063"
Oct 30 23:52:23.247135 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:23.247045    4466 generic.go:356] "Generic (PLEG): container finished" podID="5f335002-5dbf-48dc-8f24-0080043e516f" containerID="edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157" exitCode=0
Oct 30 23:52:23.247678 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:23.247124    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" event={"ID":"5f335002-5dbf-48dc-8f24-0080043e516f","Type":"ContainerDied","Data":"edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"}
Oct 30 23:52:23.247678 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:23.247606    4466 scope.go:122] "RemoveContainer" containerID="edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"
Oct 30 23:52:23.248802 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:23.248776    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:23.248901 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:23.248855    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:23.248966 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:23.248903    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:24.240846 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:24.240811    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 30 23:52:24.279876 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:24.279838    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" event={"ID":"5f335002-5dbf-48dc-8f24-0080043e516f","Type":"ContainerStarted","Data":"aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1"}
Oct 30 23:52:24.565743 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:24.564983    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:52:24.573658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:24.571796    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 23:52:24.573658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:24.571863    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 23:52:25.565124 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:25.565080    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 23:52:25.569957 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:25.569914    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 23:52:26.565658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:26.565073    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 23:52:26.565658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:26.565143    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 23:52:27.566079 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:27.566028    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 23:52:27.566682 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:27.566647    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 23:52:28.120800 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:28.120764    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:52:28.279841 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:28.279804    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 30 23:52:28.565716 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:28.565056    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 23:52:28.565716 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:28.565120    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 23:52:29.360679 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.360621    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"]
Oct 30 23:52:29.364353 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.364325    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.413919 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.413881    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-fe01fa-serving-cert\""
Oct 30 23:52:29.462594 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.462555    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"]
Oct 30 23:52:29.465962 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.465932    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.466121 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.465987    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.466189 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.466164    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.466270 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.466256    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjm8\" (UniqueName: \"kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.567260 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.567226    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.567447 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.567285    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.567447 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.567347    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.567447 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:29.567382    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-serving-cert: secret "maas-api-e2e-worker-w5-fe01fa-serving-cert" not found
Oct 30 23:52:29.567447 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.567398    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjm8\" (UniqueName: \"kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.567687 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:29.567453    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls podName:b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83 nodeName:}" failed. No retries permitted until 2026-10-30 23:52:30.067430365 +0000 UTC m=+655.357978678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls") pod "maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" (UID: "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83") : secret "maas-api-e2e-worker-w5-fe01fa-serving-cert" not found
Oct 30 23:52:29.568472 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.568439    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.570138 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.570110    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:29.570735 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.570717    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:52:29.571693 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.571675    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 30 23:52:29.627656 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:29.627564    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjm8\" (UniqueName: \"kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:30.074019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:30.073924    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:30.076493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:30.076459    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:30.280594 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:30.280551    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:30.649672 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:30.649643    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"]
Oct 30 23:52:30.652275 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:30.652246    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ac7d2a_d3a6_4f0d_b75f_a778f5e67f83.slice/crio-9f05963928eb668508354dd485b544eb311285d2448462533155539cbfe9a9ea WatchSource:0}: Error finding container 9f05963928eb668508354dd485b544eb311285d2448462533155539cbfe9a9ea: Status 404 returned error can't find the container with id 9f05963928eb668508354dd485b544eb311285d2448462533155539cbfe9a9ea
Oct 30 23:52:31.315498 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:31.315462    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" event={"ID":"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83","Type":"ContainerStarted","Data":"9f05963928eb668508354dd485b544eb311285d2448462533155539cbfe9a9ea"}
Oct 30 23:52:32.321038 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.321004    4466 generic.go:356] "Generic (PLEG): container finished" podID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerID="311993b736c51d5b6e46ee8f85496f63e80df0a0dd5bee8c00208098fecf23b2" exitCode=0
Oct 30 23:52:32.321516 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.321077    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" event={"ID":"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5","Type":"ContainerDied","Data":"311993b736c51d5b6e46ee8f85496f63e80df0a0dd5bee8c00208098fecf23b2"}
Oct 30 23:52:32.321596 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.321535    4466 scope.go:122] "RemoveContainer" containerID="311993b736c51d5b6e46ee8f85496f63e80df0a0dd5bee8c00208098fecf23b2"
Oct 30 23:52:32.322815 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.322792    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" event={"ID":"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83","Type":"ContainerStarted","Data":"883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec"}
Oct 30 23:52:32.322924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.322880    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:32.322967 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.322950    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:32.323022 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.323008    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:32.323055 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.323042    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:32.557787 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:32.557713    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" podStartSLOduration=2.68609525 podStartE2EDuration="3.557699151s" totalImagesPullingTime="871.603901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:32.556827326 +0000 UTC m=+657.847375634" watchObservedRunningTime="2026-10-30 23:52:32.557699151 +0000 UTC m=+657.848247468"
Oct 30 23:52:33.349664 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:33.341214    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" event={"ID":"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5","Type":"ContainerStarted","Data":"6ba01b36ea73a9fe73fe86ef2400feb5ad6f26b9d676c71b6aa06055aeb9110e"}
Oct 30 23:52:33.611327 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:33.611227    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:52:33.611495 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:33.611479    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 23:52:33.611547 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:33.611527    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 23:52:34.612412 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:34.612374    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 23:52:34.612859 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:34.612429    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 23:52:35.612078 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:35.612034    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 23:52:35.612284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:35.612097    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 23:52:36.611573 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:36.611530    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 23:52:36.612015 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:36.611592    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 23:52:37.612145 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:37.612103    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 23:52:37.612551 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:37.612170    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 23:52:38.318757 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:38.318714    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:52:38.363886 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.363857    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 30 23:52:38.616647 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.616602    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:52:38.617046 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.616975    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:52:38.617594 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.617578    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 30 23:52:38.764907 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.764872    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"]
Oct 30 23:52:38.772266 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.769136    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.819073 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.819036    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-46b586-serving-cert\""
Oct 30 23:52:38.826114 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.826083    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"]
Oct 30 23:52:38.894332 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.894224    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx66j\" (UniqueName: \"kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.894332 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.894293    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.894564 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.894402    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.894564 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.894438    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.996060 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.996016    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.996259 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.996079    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.996259 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.996149    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx66j\" (UniqueName: \"kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.996259 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.996189    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.996429 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:38.996342    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-serving-cert: secret "maas-api-e2e-worker-w0-46b586-serving-cert" not found
Oct 30 23:52:38.996429 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:52:38.996423    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls podName:08ba4b2d-ab49-4b55-8295-de4becb4f530 nodeName:}" failed. No retries permitted until 2026-10-30 23:52:39.496402082 +0000 UTC m=+664.786950395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls") pod "maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" (UID: "08ba4b2d-ab49-4b55-8295-de4becb4f530") : secret "maas-api-e2e-worker-w0-46b586-serving-cert" not found
Oct 30 23:52:38.996799 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.996777    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:38.998391 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:38.998372    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:39.058311 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:39.058270    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx66j\" (UniqueName: \"kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:39.502716 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:39.502674    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:39.505185 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:39.505159    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46b586-59d64b556-5kcht\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:39.684844 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:39.684808    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:40.076336 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.076309    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"]
Oct 30 23:52:40.077937 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:40.077903    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ba4b2d_ab49_4b55_8295_de4becb4f530.slice/crio-926fdae8249506a5d6a825bb4ad892932a77397da9e63b1c49f1e63433b7442a WatchSource:0}: Error finding container 926fdae8249506a5d6a825bb4ad892932a77397da9e63b1c49f1e63433b7442a: Status 404 returned error can't find the container with id 926fdae8249506a5d6a825bb4ad892932a77397da9e63b1c49f1e63433b7442a
Oct 30 23:52:40.384220 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.384183    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" event={"ID":"08ba4b2d-ab49-4b55-8295-de4becb4f530","Type":"ContainerStarted","Data":"926fdae8249506a5d6a825bb4ad892932a77397da9e63b1c49f1e63433b7442a"}
Oct 30 23:52:40.760746 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.760660    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"]
Oct 30 23:52:40.763866 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.763850    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.813300 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.813272    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-35a187-openshift-default-dockercfg-4wqw9\""
Oct 30 23:52:40.915477 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915436    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.915725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915545    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.915725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915583    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.915725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915606    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqsl\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.915725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915659    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.915725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915686    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.915725 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915720    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.916053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915763    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.916053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.915824    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:40.998647 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:40.998600    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"]
Oct 30 23:52:41.017647 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017557    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017647 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017607    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017836 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017663    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017836 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017700    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqsl\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017836 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017750    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017836 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017796    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017957 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017839    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.017957 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017898    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.018018 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.017982    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.018286 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.018256    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.018403 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.018288    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.018458 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.018398    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.018492 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.018453    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.018748 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.018721    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.020088 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.020068    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.020308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.020292    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.083266 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.083242    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.083388 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.083370    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqsl\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.377022 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.376980    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:41.391769 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.391732    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" event={"ID":"08ba4b2d-ab49-4b55-8295-de4becb4f530","Type":"ContainerStarted","Data":"74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259"}
Oct 30 23:52:41.391899 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.391807    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:41.521299 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.521241    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" podStartSLOduration=2.53299897 podStartE2EDuration="3.521226221s" totalImagesPullingTime="988.227251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:41.519544334 +0000 UTC m=+666.810092653" watchObservedRunningTime="2026-10-30 23:52:41.521226221 +0000 UTC m=+666.811774540"
Oct 30 23:52:41.820758 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.820728    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"]
Oct 30 23:52:41.822187 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:52:41.822159    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9fdfb5_5853_495b_9a47_658ce9ba9861.slice/crio-b9465baca19ed973aa61c053d0124e47d9fa4d7399bed8ccee325ab97740358b WatchSource:0}: Error finding container b9465baca19ed973aa61c053d0124e47d9fa4d7399bed8ccee325ab97740358b: Status 404 returned error can't find the container with id b9465baca19ed973aa61c053d0124e47d9fa4d7399bed8ccee325ab97740358b
Oct 30 23:52:41.824794 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.824755    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:41.824879 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.824838    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:41.824916 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:41.824881    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:42.398615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:42.398580    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" event={"ID":"cf9fdfb5-5853-495b-9a47-658ce9ba9861","Type":"ContainerStarted","Data":"ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099"}
Oct 30 23:52:42.398809 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:42.398626    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" event={"ID":"cf9fdfb5-5853-495b-9a47-658ce9ba9861","Type":"ContainerStarted","Data":"b9465baca19ed973aa61c053d0124e47d9fa4d7399bed8ccee325ab97740358b"}
Oct 30 23:52:42.522977 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:42.522918    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podStartSLOduration=2.5229017799999998 podStartE2EDuration="2.52290178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:42.522102616 +0000 UTC m=+667.812650933" watchObservedRunningTime="2026-10-30 23:52:42.52290178 +0000 UTC m=+667.813450097"
Oct 30 23:52:43.378030 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:43.377949    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:43.379744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:43.379340    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 23:52:43.379744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:43.379402    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 23:52:44.161865 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:44.161819    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:52:44.162085 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:44.161894    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:52:44.378598 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:44.378553    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 23:52:44.379034 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:44.378627    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 23:52:45.378039 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:45.378003    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 23:52:45.378250 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:45.378070    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 23:52:46.377887 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:46.377796    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 23:52:46.377887 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:46.377869    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 23:52:47.378482 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.378272    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 23:52:47.378482 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.378342    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 23:52:47.403759 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.403729    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 30 23:52:47.424663 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.424582    4466 generic.go:356] "Generic (PLEG): container finished" podID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerID="888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab" exitCode=0
Oct 30 23:52:47.424799 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.424660    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" event={"ID":"9e83d72a-ac30-48b1-86f1-83b0f29514f7","Type":"ContainerDied","Data":"888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"}
Oct 30 23:52:47.425004 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.424992    4466 scope.go:122] "RemoveContainer" containerID="888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"
Oct 30 23:52:47.426126 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.426102    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:47.426183 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.426164    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:47.426219 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:47.426192    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:48.388318 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:48.388285    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:48.389053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:48.389030    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:48.389863 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:48.389844    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 30 23:52:48.484748 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:48.484716    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" event={"ID":"9e83d72a-ac30-48b1-86f1-83b0f29514f7","Type":"ContainerStarted","Data":"186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1"}
Oct 30 23:52:49.312503 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:49.312466    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:52:49.312828 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:49.312801    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 23:52:49.312933 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:49.312863    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 23:52:50.313194 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:50.313157    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 23:52:50.313609 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:50.313212    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 23:52:51.313018 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:51.312980    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 23:52:51.313205 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:51.313048    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 23:52:52.313129 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:52.313089    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 23:52:52.313562 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:52.313159    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 23:52:53.312701 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:53.312608    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 23:52:53.312872 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:53.312722    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 23:52:54.316801 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:54.316769    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:52:54.317239 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:54.317080    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:52:54.317997 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:54.317975    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 30 23:52:56.525949 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:56.525913    4466 generic.go:356] "Generic (PLEG): container finished" podID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerID="dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857" exitCode=0
Oct 30 23:52:56.526382 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:56.525987    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" event={"ID":"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32","Type":"ContainerDied","Data":"dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"}
Oct 30 23:52:56.526422 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:56.526402    4466 scope.go:122] "RemoveContainer" containerID="dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"
Oct 30 23:52:56.527802 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:56.527776    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:56.527858 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:56.527846    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:56.527892 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:56.527876    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:52:57.555508 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:57.555466    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" event={"ID":"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32","Type":"ContainerStarted","Data":"ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd"}
Oct 30 23:52:57.939656 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:57.939033    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:52:57.952818 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:57.952769    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 23:52:57.953034 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:57.952831    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 23:52:58.120382 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:58.120351    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:52:58.950658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:58.950038    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 23:52:58.950658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:58.950117    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 23:52:59.948658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:59.941054    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 23:52:59.948658 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:52:59.941142    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 23:53:00.941798 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:00.941756    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 23:53:00.942449 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:00.942415    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 23:53:01.945099 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:01.945062    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 23:53:01.945698 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:01.945664    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 23:53:02.941341 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:02.941313    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:53:02.942367 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:02.942346    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 30 23:53:08.315756 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:53:08.315719    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:53:18.659088 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.659045    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"]
Oct 30 23:53:18.662280 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.662262    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.730257 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.730219    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"default-dockercfg-rrmnx\""
Oct 30 23:53:18.730257 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.730264    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"openshift-service-ca.crt\""
Oct 30 23:53:18.730828 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.730801    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.730958 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.730909    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6jc\" (UniqueName: \"kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.731029 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.731000    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.731109 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.731093    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.731150 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.731137    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.731281 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.731265    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.758771 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.758736    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"kube-root-ca.crt\""
Oct 30 23:53:18.758771 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.758740    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"simulator-w4-34f1ce-kserve-self-signed-certs\""
Oct 30 23:53:18.761200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.761175    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"]
Oct 30 23:53:18.832676 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.832615    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.832862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.832693    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6jc\" (UniqueName: \"kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.832862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.832723    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.832862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.832761    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.832862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.832788    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.832862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.832853    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.833100 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.833028    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.833138 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.833105    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.833210 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.833188    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.834952 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.834932    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.835215 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.835197    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.891788 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.891749    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6jc\" (UniqueName: \"kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc\") pod \"simulator-w4-34f1ce-kserve-568545fcfc-tb7zs\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") " pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:18.974261 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:18.974156    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:19.367675 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:19.361871    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"]
Oct 30 23:53:19.370804 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:53:19.370770    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ba300a_9a91_4df8_a822_9e5f21c490c2.slice/crio-bd2918a43503a34dd4777eb669bf9a1ecc920ab1e5f44ac3ef6c2ff1917344eb WatchSource:0}: Error finding container bd2918a43503a34dd4777eb669bf9a1ecc920ab1e5f44ac3ef6c2ff1917344eb: Status 404 returned error can't find the container with id bd2918a43503a34dd4777eb669bf9a1ecc920ab1e5f44ac3ef6c2ff1917344eb
Oct 30 23:53:19.675831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:19.675736    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" event={"ID":"55ba300a-9a91-4df8-a822-9e5f21c490c2","Type":"ContainerStarted","Data":"c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a"}
Oct 30 23:53:19.675831 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:19.675779    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" event={"ID":"55ba300a-9a91-4df8-a822-9e5f21c490c2","Type":"ContainerStarted","Data":"bd2918a43503a34dd4777eb669bf9a1ecc920ab1e5f44ac3ef6c2ff1917344eb"}
Oct 30 23:53:19.790569 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:19.790517    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podStartSLOduration=1.7905014339999998 podStartE2EDuration="1.790501434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:19.788907949 +0000 UTC m=+705.079456267" watchObservedRunningTime="2026-10-30 23:53:19.790501434 +0000 UTC m=+705.081049754"
Oct 30 23:53:19.974322 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:19.974244    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:28.975224 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:28.975188    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:28.987754 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:28.987727    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:29.738907 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:29.738875    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 30 23:53:35.104356 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.104319    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"]
Oct 30 23:53:35.107422 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.107397    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.203903 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.203870    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"default-dockercfg-bz6sc\""
Oct 30 23:53:35.203903 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.203877    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"simulator-w6-35a187-kserve-self-signed-certs\""
Oct 30 23:53:35.206074 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.206051    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"]
Oct 30 23:53:35.210416 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.210398    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.210466 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.210425    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72h62\" (UniqueName: \"kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.210466 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.210449    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.210542 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.210514    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.210731 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.210700    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.210767 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.210745    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.250748 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.250715    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"kube-root-ca.crt\""
Oct 30 23:53:35.250952 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.250834    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"openshift-service-ca.crt\""
Oct 30 23:53:35.311854 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.311819    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.311887    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.311907    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.311983    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.312007    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72h62\" (UniqueName: \"kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.312043    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312360 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.312334    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312360 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.312347    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.312571 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.312546    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.314270 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.314248    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.314517 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.314501    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.370854 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.370788    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72h62\" (UniqueName: \"kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62\") pod \"simulator-w6-35a187-kserve-6dc8489dd8-ljlwn\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") " pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.419128 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.419088    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:35.790263 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:35.790186    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"]
Oct 30 23:53:35.793540 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:53:35.793500    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod663be102_4fba_4766_96e2_d2825da59efe.slice/crio-a815d2f8efc48491a1339975d24319c3fa44d212a3c56de66c2bfc641012e89a WatchSource:0}: Error finding container a815d2f8efc48491a1339975d24319c3fa44d212a3c56de66c2bfc641012e89a: Status 404 returned error can't find the container with id a815d2f8efc48491a1339975d24319c3fa44d212a3c56de66c2bfc641012e89a
Oct 30 23:53:36.762992 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:36.762953    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" event={"ID":"663be102-4fba-4766-96e2-d2825da59efe","Type":"ContainerStarted","Data":"e0791b76a0ef998e1801127767b58f367d42166a09ddc37d138276c2e1cfc19d"}
Oct 30 23:53:36.763403 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:36.763001    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" event={"ID":"663be102-4fba-4766-96e2-d2825da59efe","Type":"ContainerStarted","Data":"a815d2f8efc48491a1339975d24319c3fa44d212a3c56de66c2bfc641012e89a"}
Oct 30 23:53:36.878890 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:36.878837    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podStartSLOduration=1.8788204720000001 podStartE2EDuration="1.878820472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:36.875828373 +0000 UTC m=+722.166376691" watchObservedRunningTime="2026-10-30 23:53:36.878820472 +0000 UTC m=+722.169368789"
Oct 30 23:53:37.420240 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:37.420201    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:38.317339 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:53:38.317296    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:53:45.419871 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:45.419829    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:45.432308 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:45.432274    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:45.817736 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:45.817650    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 30 23:53:55.799756 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.799721    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"]
Oct 30 23:53:55.803479 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.803457    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.830303 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.830264    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.830461 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.830393    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.830461 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.830451    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.830554 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.830475    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.830554 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.830504    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.830554 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.830534    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pml9g\" (UniqueName: \"kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.899305 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.899271    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"kube-root-ca.crt\""
Oct 30 23:53:55.899501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.899466    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"openshift-service-ca.crt\""
Oct 30 23:53:55.899501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.899482    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"default-dockercfg-ns2t6\""
Oct 30 23:53:55.899501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.899491    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"simulator-w3-986d37-kserve-self-signed-certs\""
Oct 30 23:53:55.901663 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.901621    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"]
Oct 30 23:53:55.931084 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931054    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931230 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931109    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931230 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931161    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931230 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931182    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931230 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931220    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931416 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931355    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pml9g\" (UniqueName: \"kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931520 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931501    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931620 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931540    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.931709 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.931620    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.933403 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.933384    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.933784 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.933764    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:55.989628 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:55.989598    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pml9g\" (UniqueName: \"kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g\") pod \"simulator-w3-986d37-kserve-5756b8b7d9-h59dl\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") " pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:56.116730 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:56.116687    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:53:56.494227 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:56.494206    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"]
Oct 30 23:53:56.496522 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:53:56.496494    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c41c89a_9d80_455f_9821_fa98584a88aa.slice/crio-6d101ad1aed75f7585d8e9b82722239e5ce5d11a7b89239ff6d4d04defd572b1 WatchSource:0}: Error finding container 6d101ad1aed75f7585d8e9b82722239e5ce5d11a7b89239ff6d4d04defd572b1: Status 404 returned error can't find the container with id 6d101ad1aed75f7585d8e9b82722239e5ce5d11a7b89239ff6d4d04defd572b1
Oct 30 23:53:56.862912 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:56.862876    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" event={"ID":"9c41c89a-9d80-455f-9821-fa98584a88aa","Type":"ContainerStarted","Data":"81112adbe3445e729dbfb7aedb8e53371b819de9a93bd4866a8855bd3a74624f"}
Oct 30 23:53:56.863344 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:56.862925    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" event={"ID":"9c41c89a-9d80-455f-9821-fa98584a88aa","Type":"ContainerStarted","Data":"6d101ad1aed75f7585d8e9b82722239e5ce5d11a7b89239ff6d4d04defd572b1"}
Oct 30 23:53:56.974934 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:56.974877    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podStartSLOduration=1.974861159 podStartE2EDuration="1.974861159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:56.972839539 +0000 UTC m=+742.263387859" watchObservedRunningTime="2026-10-30 23:53:56.974861159 +0000 UTC m=+742.265409476"
Oct 30 23:53:57.117681 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:53:57.117568    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:54:06.117437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:06.117400    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:54:06.129699 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:06.129663    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:54:06.931588 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:06.931558    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 30 23:54:08.317493 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:54:08.317447    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:54:27.943120 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.943029    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"]
Oct 30 23:54:27.946474 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.946451    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:27.981064 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.981025    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjfk\" (UniqueName: \"kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:27.981244 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.981128    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:27.981244 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.981214    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:27.981371 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.981274    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:27.981414 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.981368    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:27.981479 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:27.981455    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.042747 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.042713    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"premium-w4-34f1ce-kserve-self-signed-certs\""
Oct 30 23:54:28.045040 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.045018    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"]
Oct 30 23:54:28.082757 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.082720    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.082955 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.082808    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.082955 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.082891    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjfk\" (UniqueName: \"kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.082955 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.082939    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.083132 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.082987    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.083132 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.083014    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.084004 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.083677    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.084004 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.083749    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.084004 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.083958    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.086053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.086032    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.086166 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.086096    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.142312 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.142275    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjfk\" (UniqueName: \"kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk\") pod \"premium-w4-34f1ce-kserve-6bfc6b564-h27c9\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") " pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.258673 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.258564    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:28.624642 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:28.624605    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"]
Oct 30 23:54:28.626548 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:54:28.626519    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb8cd2b1_3b6e_429d_9f50_c14bd23cf52a.slice/crio-0cb55e4bf6b35592b92027f3b970af00a5b55a54290c4abaf578f5b242a16e4d WatchSource:0}: Error finding container 0cb55e4bf6b35592b92027f3b970af00a5b55a54290c4abaf578f5b242a16e4d: Status 404 returned error can't find the container with id 0cb55e4bf6b35592b92027f3b970af00a5b55a54290c4abaf578f5b242a16e4d
Oct 30 23:54:29.028568 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:29.028471    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" event={"ID":"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a","Type":"ContainerStarted","Data":"280316e7389022e3323d6ae788a1e2eef6f9224fe5f51688c5d9d9c8da9200d6"}
Oct 30 23:54:29.028568 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:29.028522    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" event={"ID":"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a","Type":"ContainerStarted","Data":"0cb55e4bf6b35592b92027f3b970af00a5b55a54290c4abaf578f5b242a16e4d"}
Oct 30 23:54:29.147251 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:29.147191    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podStartSLOduration=2.147173426 podStartE2EDuration="2.147173426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:29.139332418 +0000 UTC m=+774.429880738" watchObservedRunningTime="2026-10-30 23:54:29.147173426 +0000 UTC m=+774.437721744"
Oct 30 23:54:29.259297 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:29.259264    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:38.259719 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:38.259674    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:38.271552 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:38.271525    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:54:38.316284 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:54:38.316243    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:54:39.090721 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:54:39.090694    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 30 23:55:08.316322 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:55:08.316278    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:55:19.551539 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.551502    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"]
Oct 30 23:55:19.555265 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.555245    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.654076 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.654040    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.654248 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.654082    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslst\" (UniqueName: \"kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.654248 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.654188    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.654371 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.654265    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.654371 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.654330    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.654473 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.654419    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.656115 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.656092    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"default-dockercfg-kv2s5\""
Oct 30 23:55:19.656226 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.656208    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"premium-w5-fe01fa-kserve-self-signed-certs\""
Oct 30 23:55:19.656285 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.656267    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"openshift-service-ca.crt\""
Oct 30 23:55:19.656334 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.656288    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"kube-root-ca.crt\""
Oct 30 23:55:19.658701 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.658680    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"]
Oct 30 23:55:19.755883 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.755845    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.755909    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756012    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756059 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756040    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslst\" (UniqueName: \"kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756166 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756098    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756166 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756149    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756319 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756300    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756368 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756334    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.756451 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.756437    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.758390 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.758367    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.758505 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.758488    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.814909 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.814834    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslst\" (UniqueName: \"kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst\") pod \"premium-w5-fe01fa-kserve-55ddb999bd-wzd2s\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") " pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:19.866598 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:19.866567    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:20.238874 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:20.238844    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"]
Oct 30 23:55:20.240799 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:55:20.240769    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26f1d7e9_1e18_4204_9e6e_d86b831c4bd0.slice/crio-fc317c03d0638d36df5d63d2e25867b8c1c8ae4941800154885364da6505eaf9 WatchSource:0}: Error finding container fc317c03d0638d36df5d63d2e25867b8c1c8ae4941800154885364da6505eaf9: Status 404 returned error can't find the container with id fc317c03d0638d36df5d63d2e25867b8c1c8ae4941800154885364da6505eaf9
Oct 30 23:55:20.292361 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:20.292325    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" event={"ID":"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0","Type":"ContainerStarted","Data":"fc317c03d0638d36df5d63d2e25867b8c1c8ae4941800154885364da6505eaf9"}
Oct 30 23:55:21.299019 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:21.298985    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" event={"ID":"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0","Type":"ContainerStarted","Data":"ea10389c3c99be63c84b8a2afea0a8317d2dac9dcaf3037ee87f705d06ac1df2"}
Oct 30 23:55:21.411545 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:21.411488    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podStartSLOduration=2.411471875 podStartE2EDuration="2.411471875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:21.409960495 +0000 UTC m=+826.700508811" watchObservedRunningTime="2026-10-30 23:55:21.411471875 +0000 UTC m=+826.702020193"
Oct 30 23:55:21.867196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:21.867155    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:29.867042 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:29.867005    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:29.879285 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:29.879257    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:30.351107 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:55:30.351080    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 30 23:55:38.316739 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:55:38.316700    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:56:08.319903 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:56:08.319860    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:56:38.317125 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:56:38.317089    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:57:08.316234 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:57:08.316191    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:57:21.063648 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.063550    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"]
Oct 30 23:57:21.067427 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.067405    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.163242 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163209    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"kube-root-ca.crt\""
Oct 30 23:57:21.163437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163272    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"openshift-service-ca.crt\""
Oct 30 23:57:21.163437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163284    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"default-dockercfg-fst44\""
Oct 30 23:57:21.163437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163415    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"distinct-w0-46b586-kserve-self-signed-certs\""
Oct 30 23:57:21.163615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163591    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.163806 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163789    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.163879 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163828    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.163879 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163858    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8dx\" (UniqueName: \"kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.163978 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163953    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.164037 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.163998    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.165529 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.165510    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"]
Oct 30 23:57:21.264881 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.264851    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.264881 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.264888    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265104 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.264905    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8dx\" (UniqueName: \"kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265206 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.265187    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265267 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.265250    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265342 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.265323    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265464 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.265446    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265616 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.265599    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.265694 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.265676    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.267508 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.267482    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.267619 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.267562    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.324100 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.324034    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8dx\" (UniqueName: \"kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx\") pod \"distinct-w0-46b586-kserve-6c5777f6bd-89b8x\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.378783 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.378749    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:21.745227 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.745193    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"]
Oct 30 23:57:21.747705 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:57:21.747672    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3bb64b_08ff_4946_bd36_14178156cc8d.slice/crio-99f43af7dd1cd210ec34333cc09fbdb1f55b7f2ac1646a4ba02acef8e941e8fb WatchSource:0}: Error finding container 99f43af7dd1cd210ec34333cc09fbdb1f55b7f2ac1646a4ba02acef8e941e8fb: Status 404 returned error can't find the container with id 99f43af7dd1cd210ec34333cc09fbdb1f55b7f2ac1646a4ba02acef8e941e8fb
Oct 30 23:57:21.942877 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.942839    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" event={"ID":"7b3bb64b-08ff-4946-bd36-14178156cc8d","Type":"ContainerStarted","Data":"c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e"}
Oct 30 23:57:21.942877 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:21.942882    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" event={"ID":"7b3bb64b-08ff-4946-bd36-14178156cc8d","Type":"ContainerStarted","Data":"99f43af7dd1cd210ec34333cc09fbdb1f55b7f2ac1646a4ba02acef8e941e8fb"}
Oct 30 23:57:22.056692 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:22.056575    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podStartSLOduration=1.05655725 podStartE2EDuration="1.05655725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:22.054558759 +0000 UTC m=+947.345107077" watchObservedRunningTime="2026-10-30 23:57:22.05655725 +0000 UTC m=+947.347105567"
Oct 30 23:57:22.379140 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:22.379105    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:31.378864 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:31.378828    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:31.391245 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:31.391217    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:32.015733 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:32.015703    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 30 23:57:38.316363 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:57:38.316319    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:57:43.142493 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.142455    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"]
Oct 30 23:57:43.150009 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.149976    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.252152 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.252115    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"]
Oct 30 23:57:43.336200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336163    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336207    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336489 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336340    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336489 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336381    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336489 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336465    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336622 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336520    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336622 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336542    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336622 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336599    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.336751 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.336670    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv48g\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438138 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438041    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438138 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438103    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv48g\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438138 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438127    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438151    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438202    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438231    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438289    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438338    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438437 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438369    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438833 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438705    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.438833 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.438745    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.439055 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.439033    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.439131 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.439032    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.439397 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.439373    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.440708 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.440685    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.440783 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.440768    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.495948 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.495917    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.496089 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.496050    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv48g\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:43.762900 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:43.762810    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:44.181521 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:44.181492    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"]
Oct 30 23:57:44.183726 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:57:44.183699    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7147a543_9acb_4d97_b8cd_1f95da39b70b.slice/crio-9b2eeb205334f4075f03271c9c651f8ea4cb2f4e95e1fad2e38e7fea135bff5f WatchSource:0}: Error finding container 9b2eeb205334f4075f03271c9c651f8ea4cb2f4e95e1fad2e38e7fea135bff5f: Status 404 returned error can't find the container with id 9b2eeb205334f4075f03271c9c651f8ea4cb2f4e95e1fad2e38e7fea135bff5f
Oct 30 23:57:44.186026 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:44.185991    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:57:44.186107 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:44.186060    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:57:44.186107 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:44.186092    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:57:45.073546 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.073510    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" event={"ID":"7147a543-9acb-4d97-b8cd-1f95da39b70b","Type":"ContainerStarted","Data":"f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c"}
Oct 30 23:57:45.073790 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.073578    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" event={"ID":"7147a543-9acb-4d97-b8cd-1f95da39b70b","Type":"ContainerStarted","Data":"9b2eeb205334f4075f03271c9c651f8ea4cb2f4e95e1fad2e38e7fea135bff5f"}
Oct 30 23:57:45.173963 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.173927    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"]
Oct 30 23:57:45.177696 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.177675    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.257620 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.257585    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"]
Oct 30 23:57:45.298443 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.298389    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podStartSLOduration=2.298372535 podStartE2EDuration="2.298372535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:45.273426618 +0000 UTC m=+970.563974974" watchObservedRunningTime="2026-10-30 23:57:45.298372535 +0000 UTC m=+970.588920852"
Oct 30 23:57:45.364722 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.364692    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.364922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.364733    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.364922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.364759    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.364922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.364865    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.365035 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.364919    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.365035 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.364967    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.365035 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.365030    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.365140 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.365047    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.365176 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.365145    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvz4s\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.465747 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.465712    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.465747 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.465756    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.465985 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.465773    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.465985 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.465826    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.465985 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.465875    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.465985 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.465899    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466185 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466080    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466185 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466121    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466303 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466190    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvz4s\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466303 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466253    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466389 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466298    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466462 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466438    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466497 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466482    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.466802 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.466779    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.468335 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.468314    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.468473 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.468455    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.524025 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.523993    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.524229 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.524210    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvz4s\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s\") pod \"maas-default-gateway-openshift-default-8f6f87f59-vt4mg\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:45.764067 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.763980    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:45.765515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.765489    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 23:57:45.765574 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.765545    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 23:57:45.792458 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:45.792429    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:46.253686 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:46.253442    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"]
Oct 30 23:57:46.257204 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:46.257169    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:57:46.257293 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:46.257257    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:57:46.257336 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:46.257302    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:57:46.763991 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:46.763956    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 23:57:46.764500 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:46.764017    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 23:57:47.085646 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.085537    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" event={"ID":"39910e80-d7fb-488b-a65b-246e3f74bc90","Type":"ContainerStarted","Data":"22c3bc404b6e6b7eaf3b658f6c8c2de0ef57ed37d114b42be716c42eacdd9858"}
Oct 30 23:57:47.085646 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.085593    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" event={"ID":"39910e80-d7fb-488b-a65b-246e3f74bc90","Type":"ContainerStarted","Data":"e6dade7eac739147438300ac299d7416b8fc04293336570ac6f3ae80635714cb"}
Oct 30 23:57:47.207356 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.207303    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podStartSLOduration=2.207287162 podStartE2EDuration="2.207287162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:47.206148235 +0000 UTC m=+972.496696553" watchObservedRunningTime="2026-10-30 23:57:47.207287162 +0000 UTC m=+972.497835478"
Oct 30 23:57:47.774927 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.774893    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 23:57:47.775497 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.775466    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 23:57:47.793509 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.793444    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:47.794538 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.794509    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:47.794671 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:47.794566    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:48.769659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:48.769069    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 23:57:48.769659 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:48.769132    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 23:57:48.795055 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:48.795020    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:48.795453 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:48.795073    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:49.765026 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:49.764992    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 23:57:49.765319 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:49.765297    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 23:57:49.792881 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:49.792839    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:49.793070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:49.792913    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:50.793020 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:50.792982    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:50.793393 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:50.793049    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:51.060196 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:51.060108    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:51.061085 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:51.060996    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:51.062132 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:51.062103    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 30 23:57:51.793390 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:51.793353    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:51.793944 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:51.793421    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:52.793452 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:52.793422    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:52.793869 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:52.793479    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:53.793791 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:53.793752    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-vt4mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 23:57:53.794323 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:53.793813    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 23:57:55.157809 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:55.157770    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:55.159046 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:55.158992    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:57:55.160227 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:57:55.160207    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 30 23:58:08.316209 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:58:08.316167    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:58:13.750290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.750252    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"]
Oct 30 23:58:13.755249 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.755224    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.817870 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.817828    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"]
Oct 30 23:58:13.872173 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.871397    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872365 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872269    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872365 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872335    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872498 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872418    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872498 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872472    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872610 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872526    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrm2\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872707 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872620    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872778 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872721    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.872832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.872779    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.973707 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973657    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.973922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973725    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.973922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973786    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.973922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973817    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.973922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973881    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.973922 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973920    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974203 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973963    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrm2\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974203 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.973987    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974203 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.974053    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974366 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.974257    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974366 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.974273    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.974721    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.974897 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.974769    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.975000 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.974940    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.977118 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.977089    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:13.977306 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:13.977289    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:14.033778 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.033704    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:14.035744 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.035723    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrm2\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:14.075498 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.075457    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:14.497056 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.497027    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"]
Oct 30 23:58:14.498872 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:58:14.498841    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcee6fe_3f93_499a_aca5_406342361197.slice/crio-35f9b487fba6a809b5adcdaf571577fabafe63fcef86f7fa0711b6b6f4b47221 WatchSource:0}: Error finding container 35f9b487fba6a809b5adcdaf571577fabafe63fcef86f7fa0711b6b6f4b47221: Status 404 returned error can't find the container with id 35f9b487fba6a809b5adcdaf571577fabafe63fcef86f7fa0711b6b6f4b47221
Oct 30 23:58:14.501199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.501169    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:58:14.501254 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.501234    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:58:14.501289 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:14.501261    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:58:15.260339 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:15.260304    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" event={"ID":"8bcee6fe-3f93-499a-aca5-406342361197","Type":"ContainerStarted","Data":"8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0"}
Oct 30 23:58:15.260800 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:15.260364    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" event={"ID":"8bcee6fe-3f93-499a-aca5-406342361197","Type":"ContainerStarted","Data":"35f9b487fba6a809b5adcdaf571577fabafe63fcef86f7fa0711b6b6f4b47221"}
Oct 30 23:58:15.468445 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:15.468384    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podStartSLOduration=2.468368816 podStartE2EDuration="2.468368816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:15.421318776 +0000 UTC m=+1000.711867104" watchObservedRunningTime="2026-10-30 23:58:15.468368816 +0000 UTC m=+1000.758917130"
Oct 30 23:58:16.076289 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:16.076257    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:16.077695 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:16.077665    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 23:58:16.077818 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:16.077726    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 23:58:17.076459 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:17.076414    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 23:58:17.076888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:17.076479    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 23:58:18.076808 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.076771    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 23:58:18.077200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.076838    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 23:58:18.405515 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.405477    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"]
Oct 30 23:58:18.409888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.409863    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.466577 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.466521    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"]
Oct 30 23:58:18.508475 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.508444    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"distinct-w3-986d37-kserve-self-signed-certs\""
Oct 30 23:58:18.542670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.528147    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.542670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.528223    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.542670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.528278    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.542670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.528316    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.542670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.528369    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qbf\" (UniqueName: \"kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.542670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.528461    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.640341    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.640476    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.640522    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.640572    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.640608    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.640670    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qbf\" (UniqueName: \"kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.641505    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.641777    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.642859    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.659762 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.649112    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.660546 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.660396    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.698501 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.698463    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qbf\" (UniqueName: \"kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf\") pod \"distinct-w3-986d37-kserve-646749ff-wh9mb\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:18.722832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:18.722801    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:19.076250 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.076161    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 23:58:19.076250 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.076226    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 23:58:19.121719 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.121695    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"]
Oct 30 23:58:19.123607 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:58:19.123566    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e56b66_19c9_45b6_b6eb_f12ed36f5b6f.slice/crio-334181cd96565c11ec9e003aa7300698a6051075007f881d2303537b9435dcdb WatchSource:0}: Error finding container 334181cd96565c11ec9e003aa7300698a6051075007f881d2303537b9435dcdb: Status 404 returned error can't find the container with id 334181cd96565c11ec9e003aa7300698a6051075007f881d2303537b9435dcdb
Oct 30 23:58:19.283667 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.283623    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" event={"ID":"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f","Type":"ContainerStarted","Data":"42c9937523f02d37d3c548222d4daa70796d7ebc3fd50ab74021069ce7047113"}
Oct 30 23:58:19.283852 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.283677    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" event={"ID":"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f","Type":"ContainerStarted","Data":"334181cd96565c11ec9e003aa7300698a6051075007f881d2303537b9435dcdb"}
Oct 30 23:58:19.414049 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.413994    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podStartSLOduration=1.413977376 podStartE2EDuration="1.413977376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:19.41134998 +0000 UTC m=+1004.701898297" watchObservedRunningTime="2026-10-30 23:58:19.413977376 +0000 UTC m=+1004.704525694"
Oct 30 23:58:19.723752 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:19.723666    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:20.076862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:20.076777    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 23:58:20.076862 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:20.076832    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 23:58:21.409426 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:21.409389    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:21.409993 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:21.409968    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:21.410848 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:21.410822    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 30 23:58:24.180869 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.180839    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"]
Oct 30 23:58:24.184475 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.184456    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.239823 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.239789    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.240004 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.239839    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.240004 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.239979    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.240110 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.240037    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.240159 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.240110    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.240205 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.240170    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxsc\" (UniqueName: \"kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.280768 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.280733    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"distinct-w5-fe01fa-kserve-self-signed-certs\""
Oct 30 23:58:24.283458 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.283434    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"]
Oct 30 23:58:24.342144 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342112    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.342394 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342149    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.342394 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342247    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.342394 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342301    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.342556 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342398    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.342556 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342461    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxsc\" (UniqueName: \"kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.342826 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.342799    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.343106 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.343080    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.343239 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.343107    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.345367 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.345338    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.347034 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.347009    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.400619 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.400587    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxsc\" (UniqueName: \"kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc\") pod \"distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") " pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.496977 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.496894    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:24.866802 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:24.866765    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"]
Oct 30 23:58:25.317753 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:25.317661    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" event={"ID":"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa","Type":"ContainerStarted","Data":"638bfe5cff38ad058ca4cb0d1af29aa88397b7cceb08c19f5bbeba4424a4832f"}
Oct 30 23:58:25.317753 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:25.317719    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" event={"ID":"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa","Type":"ContainerStarted","Data":"12e3f20f2fd28a88f2fa7aa56b8ed1c73316e3982663ff04fb40a88f8b1100e2"}
Oct 30 23:58:25.430935 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:25.430866    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podStartSLOduration=1.430846674 podStartE2EDuration="1.430846674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:25.429150209 +0000 UTC m=+1010.719698527" watchObservedRunningTime="2026-10-30 23:58:25.430846674 +0000 UTC m=+1010.721394992"
Oct 30 23:58:25.497338 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:25.497300    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:28.723192 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:28.723103    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:28.735977 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:28.735942    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:29.348619 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:29.348592    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 30 23:58:34.497787 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:34.497745    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:34.510322 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:34.510296    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:35.388478 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:35.388444    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 30 23:58:38.316801 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:58:38.316756    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:58:43.650754 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.650718    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"]
Oct 30 23:58:43.654783 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.654762    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.750566 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.750536    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"distinct-w6-35a187-kserve-self-signed-certs\""
Oct 30 23:58:43.753205 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.753184    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"]
Oct 30 23:58:43.782889 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.782854    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.783116 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.782898    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.783116 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.782981    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwsj\" (UniqueName: \"kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.783116 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.783050    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.783116 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.783088    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.783250 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.783172    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884127 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884092    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884127 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884132    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884385 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884168    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwsj\" (UniqueName: \"kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884385 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884230    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884385 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884261    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884385 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884305    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884592 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884494    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884670 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884611    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.884729 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.884687    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.886451 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.886427    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.886857 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.886841    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.942098 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.942023    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwsj\" (UniqueName: \"kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj\") pod \"distinct-w6-35a187-kserve-846b7dc696-rq8ds\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") " pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:43.966867 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:43.966833    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:44.335743 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:44.335716    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"]
Oct 30 23:58:44.338074 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:58:44.338045    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18b27e63_21b5_4321_b826_909e9d215a4d.slice/crio-8970739d310fd61929d0fe810652267fe5a7ac07da555177808f1b0cc38893d4 WatchSource:0}: Error finding container 8970739d310fd61929d0fe810652267fe5a7ac07da555177808f1b0cc38893d4: Status 404 returned error can't find the container with id 8970739d310fd61929d0fe810652267fe5a7ac07da555177808f1b0cc38893d4
Oct 30 23:58:44.424731 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:44.424698    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" event={"ID":"18b27e63-21b5-4321-b826-909e9d215a4d","Type":"ContainerStarted","Data":"c59b1cf66983594d2c5ebeeaf63a6cf0b04dd042bad10d5270499e21b935fa23"}
Oct 30 23:58:44.424865 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:44.424742    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" event={"ID":"18b27e63-21b5-4321-b826-909e9d215a4d","Type":"ContainerStarted","Data":"8970739d310fd61929d0fe810652267fe5a7ac07da555177808f1b0cc38893d4"}
Oct 30 23:58:44.545807 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:44.545754    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podStartSLOduration=1.5457357680000001 podStartE2EDuration="1.545735768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:44.544510054 +0000 UTC m=+1029.835058372" watchObservedRunningTime="2026-10-30 23:58:44.545735768 +0000 UTC m=+1029.836284085"
Oct 30 23:58:44.967953 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:44.967918    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:53.967968 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:53.967920    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:53.980569 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:53.980540    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:54.485763 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:54.485731    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 30 23:58:57.985401 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:57.985365    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"]
Oct 30 23:58:57.989329 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:57.989310    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.039143 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.039116    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-986d37-openshift-default-dockercfg-rm9vh\""
Oct 30 23:58:58.059802 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.059767    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.059915 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.059807    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.059956 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.059907    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.059990 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.059971    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.060023 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.060006    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.060194 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.060170    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.060256 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.060244    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.060290 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.060279    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.060323 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.060299    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hssc\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.090237 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.090194    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"]
Oct 30 23:58:58.161990 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.161952    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162187 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.161998    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162187 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162028    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hssc\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162187 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162095    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162187 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162119    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162187 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162164    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162434 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162247    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162434 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162298    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162434 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162389    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162575 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162552    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162661 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162589    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162801 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162775    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.162949 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.162929    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.163117 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.163100    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.164444 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.164415    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.164555 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.164489    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.219981 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.219944    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.220147 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.220036    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hssc\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.302731 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.302612    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:58:58.724891 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.724858    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"]
Oct 30 23:58:58.726344 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:58:58.726315    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0be6323_9b09_4367_a147_c5e83b1d4600.slice/crio-6d32bce3bec6925d1ed61ce4f53b6c8a0248cf06523de6da2a5c58dcb717b897 WatchSource:0}: Error finding container 6d32bce3bec6925d1ed61ce4f53b6c8a0248cf06523de6da2a5c58dcb717b897: Status 404 returned error can't find the container with id 6d32bce3bec6925d1ed61ce4f53b6c8a0248cf06523de6da2a5c58dcb717b897
Oct 30 23:58:58.729105 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.729074    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:58:58.729208 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.729151    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:58:58.729208 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:58.729196    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:58:59.504917 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.504876    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" event={"ID":"e0be6323-9b09-4367-a147-c5e83b1d4600","Type":"ContainerStarted","Data":"6e720e5c2a12303530d8539a230519761a94a89992d3a7e3ce13c030ae5a4ce8"}
Oct 30 23:58:59.507116 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.504949    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" event={"ID":"e0be6323-9b09-4367-a147-c5e83b1d4600","Type":"ContainerStarted","Data":"6d32bce3bec6925d1ed61ce4f53b6c8a0248cf06523de6da2a5c58dcb717b897"}
Oct 30 23:58:59.625792 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.625738    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podStartSLOduration=2.625721579 podStartE2EDuration="2.625721579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:59.624726403 +0000 UTC m=+1044.915274723" watchObservedRunningTime="2026-10-30 23:58:59.625721579 +0000 UTC m=+1044.916269896"
Oct 30 23:58:59.700400 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.700368    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"]
Oct 30 23:58:59.704317 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.704296    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.753582 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.753547    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5c3374-openshift-default-dockercfg-jbr44\""
Oct 30 23:58:59.763150 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.763050    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"]
Oct 30 23:58:59.783314 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783281    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783506 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783327    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783506 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783420    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783506 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783475    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783516    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783687 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783625    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783790 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783713    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783842 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783797    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2zm\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.783893 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.783867    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.885570 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885536    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.885779 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885580    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.885779 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885673    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.885928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885845    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.885928 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885901    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886029 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885938    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886029 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.885971    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886029 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886014    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2zm\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886059    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886199 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886070    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886312 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886247    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886504 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886481    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886577 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886506    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.886684 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.886664    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.888142 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.888123    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.888420 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.888403    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.957352 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.954042    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:58:59.964053 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:58:59.964024    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2zm\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:59:00.020701 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.020611    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:59:00.303253 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.303168    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:59:00.305264 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.305230    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:59:00.305415 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.305305    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:59:00.466671 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.466627    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"]
Oct 30 23:59:00.468951 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:59:00.468918    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086adb9a_4bca_44a8_ad31_051f0219b802.slice/crio-a2f624dc235b1ba2cd0875cfa7d573fffc5a66b461e4345223d77b1188f8ea10 WatchSource:0}: Error finding container a2f624dc235b1ba2cd0875cfa7d573fffc5a66b461e4345223d77b1188f8ea10: Status 404 returned error can't find the container with id a2f624dc235b1ba2cd0875cfa7d573fffc5a66b461e4345223d77b1188f8ea10
Oct 30 23:59:00.471547 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.471511    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:00.471690 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.471601    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:00.471690 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.471653    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:00.512224 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:00.512189    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" event={"ID":"086adb9a-4bca-44a8-ad31-051f0219b802","Type":"ContainerStarted","Data":"a2f624dc235b1ba2cd0875cfa7d573fffc5a66b461e4345223d77b1188f8ea10"}
Oct 30 23:59:01.303626 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:01.303595    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:59:01.303830 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:01.303679    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:59:01.519904 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:01.519869    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" event={"ID":"086adb9a-4bca-44a8-ad31-051f0219b802","Type":"ContainerStarted","Data":"fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365"}
Oct 30 23:59:01.690932 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:01.690140    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podStartSLOduration=2.690119628 podStartE2EDuration="2.690119628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:01.684332 +0000 UTC m=+1046.974880319" watchObservedRunningTime="2026-10-30 23:59:01.690119628 +0000 UTC m=+1046.980667947"
Oct 30 23:59:02.021860 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:02.021770    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:59:02.023380 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:02.023354    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:59:02.023508 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:02.023397    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:59:02.303519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:02.303427    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:59:02.303519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:02.303483    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:59:03.021024 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:03.020986    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:59:03.021431 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:03.021045    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:59:03.303512 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:03.303420    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:59:03.303512 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:03.303483    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:59:04.021696 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:04.021627    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:59:04.022107 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:04.021717    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:59:04.303464 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:04.303369    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:59:04.303464 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:04.303429    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:59:05.021099 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:05.021057    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:59:05.021318 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:05.021119    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:59:05.303795 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:05.303693    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:59:05.303795 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:05.303747    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:59:06.021926 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:06.021892    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:59:06.022110 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:06.021951    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:59:06.308038 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:06.307957    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:59:06.308410 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:06.308252    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:59:06.308886 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:06.308871    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 30 23:59:07.021013 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:07.020981    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:59:07.021198 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:07.021034    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:59:08.206077 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:08.206041    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:59:08.206903 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:08.206872    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:59:08.207227 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:08.207208    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 30 23:59:08.319535 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:59:08.319502    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:59:28.967136 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:28.967099    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"]
Oct 30 23:59:28.970990 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:28.970969    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004418 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004381    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004580 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004432    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004625 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004572    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004697 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004681    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpk9\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004741 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004711    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004741 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004737    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004837 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004788    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004890 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004848    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.004941 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.004908    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106200 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106159    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106401 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106224    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106401 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106298    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106410    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpk9\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106454    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106519 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106500    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106712 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106530    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106712 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106574    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106712 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106662    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106712 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106686    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.106950 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106888    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.107007 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106955    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.107007 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.106979    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.107110 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.107002    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.108569 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.108550    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.108908 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.108891    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.164832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.164797    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.164832 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.164829    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpk9\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.172768 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.172698    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"]
Oct 30 23:59:29.284797 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.284706    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:29.706552 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.706513    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"]
Oct 30 23:59:29.708005 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:59:29.707978    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200386c3_07db_4689_ba2d_ea12c713b4df.slice/crio-830276ad5d7242036147e630c2bd58ebaa93186d73804deb1292ec15902402d8 WatchSource:0}: Error finding container 830276ad5d7242036147e630c2bd58ebaa93186d73804deb1292ec15902402d8: Status 404 returned error can't find the container with id 830276ad5d7242036147e630c2bd58ebaa93186d73804deb1292ec15902402d8
Oct 30 23:59:29.710724 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.710684    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:29.710824 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.710787    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:29.710864 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.710834    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:29.715869 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:29.715841    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" event={"ID":"200386c3-07db-4689-ba2d-ea12c713b4df","Type":"ContainerStarted","Data":"830276ad5d7242036147e630c2bd58ebaa93186d73804deb1292ec15902402d8"}
Oct 30 23:59:30.722420 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:30.722383    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" event={"ID":"200386c3-07db-4689-ba2d-ea12c713b4df","Type":"ContainerStarted","Data":"4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2"}
Oct 30 23:59:30.887851 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:30.887791    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podStartSLOduration=2.887773912 podStartE2EDuration="2.887773912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:30.884843535 +0000 UTC m=+1076.175391881" watchObservedRunningTime="2026-10-30 23:59:30.887773912 +0000 UTC m=+1076.178322229"
Oct 30 23:59:31.285622 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:31.285592    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:31.287080 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:31.287059    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:59:31.287138 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:31.287109    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:59:32.285825 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:32.285788    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:59:32.286231 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:32.285841    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:59:33.285096 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:33.285055    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:59:33.285261 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:33.285129    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:59:34.286070 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:34.286031    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:59:34.286483 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:34.286092    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:59:35.285351 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:35.285314    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:59:35.285544 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:35.285377    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:59:36.529924 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:36.529888    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:36.530811 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:36.530783    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:36.532176 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:36.532154    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 30 23:59:38.316473 ip-10-0-131-123 kubenswrapper[4466]: E1030 23:59:38.316433    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 30 23:59:43.649529 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.649482    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"]
Oct 30 23:59:43.654666 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.654612    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.752888 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.752855    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"]
Oct 30 23:59:43.818507 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818466    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.818707 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818529    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.818707 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818584    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvjj\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.818707 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818615    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.818821 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818811    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.818854 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818836    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.818941 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.818925    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.819033 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.819013    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.819095 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.819071    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920189 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920100    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvjj\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920189 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920141    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920424 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920235    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920424 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920260    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920424 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920319    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920424 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920365    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920424 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920391    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920702 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920494    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920702 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920535    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920820 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920797    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920998 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920975    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.920998 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.920986    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.921153 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.921131    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.921207 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.921173    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.922848 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.922827    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.922954 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.922930    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.978360 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.978324    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:43.978511 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:43.978434    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvjj\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:44.267842 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.267744    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:44.694100 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.694072    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"]
Oct 30 23:59:44.695998 ip-10-0-131-123 kubenswrapper[4466]: W1030 23:59:44.695970    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdfe64d_5157_4b10_a0c2_dc21f29c1a41.slice/crio-0774ce36a638a4c9bb5b1705816ff4788cfff630be226fc98a2dca6ffbaa57fb WatchSource:0}: Error finding container 0774ce36a638a4c9bb5b1705816ff4788cfff630be226fc98a2dca6ffbaa57fb: Status 404 returned error can't find the container with id 0774ce36a638a4c9bb5b1705816ff4788cfff630be226fc98a2dca6ffbaa57fb
Oct 30 23:59:44.698560 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.698529    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:44.698672 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.698604    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:44.698672 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.698652    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:59:44.861654 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.855505    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" event={"ID":"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41","Type":"ContainerStarted","Data":"09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3"}
Oct 30 23:59:44.861654 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.855565    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" event={"ID":"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41","Type":"ContainerStarted","Data":"0774ce36a638a4c9bb5b1705816ff4788cfff630be226fc98a2dca6ffbaa57fb"}
Oct 30 23:59:44.976083 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:44.975958    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podStartSLOduration=1.9759365020000002 podStartE2EDuration="1.975936502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:44.971087968 +0000 UTC m=+1090.261636286" watchObservedRunningTime="2026-10-30 23:59:44.975936502 +0000 UTC m=+1090.266484820"
Oct 30 23:59:45.268541 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:45.268449    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:45.270344 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:45.270311    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:45.270497 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:45.270360    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:46.271150 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:46.271101    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:46.271543 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:46.271174    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:47.268505 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:47.268467    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:47.268735 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:47.268526    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:48.268131 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:48.268095    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:48.268525 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:48.268158    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:49.268900 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:49.268865    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:49.269397 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:49.268938    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:50.268935 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:50.268888    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:50.269354 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:50.268955    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:51.268568 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:51.268537    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:59:51.268773 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:51.268591    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 23:59:52.272958 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:52.272929    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:52.273362 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:52.273207    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:52.273980 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:52.273958    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 30 23:59:59.584845 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.584814    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"]
Oct 30 23:59:59.588472 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.588454    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659557 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659519    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659751 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659644    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659751 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659694    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659751 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659711    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrn6\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659867 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659851    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659901 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659881    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659936 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659901    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.659969 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659931    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.660010 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.659994    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761398 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761353    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761427    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761456    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrn6\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761546    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761615 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761587    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761895 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761622    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761895 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761679    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761895 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761743    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.761895 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.761880    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.762191 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.762163    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.762284 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.762230    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.762345 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.762311    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.762410 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.762363    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.762739 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.762715    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.764277 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.764251    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.764382 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.764365    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.788851 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.788810    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"]
Oct 30 23:59:59.820159 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.820099    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.820159 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.820141    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrn6\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 30 23:59:59.902436 ip-10-0-131-123 kubenswrapper[4466]: I1030 23:59:59.902392    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:00:00.336332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:00.336299    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"]
Oct 31 00:00:00.337322 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:00:00.337289    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd37248_4e95_4847_a9b7_07d94e99aa4a.slice/crio-d6366bdde5feb9ed0672b994d25298c82c82babc53adaefe43465b234f4f5003 WatchSource:0}: Error finding container d6366bdde5feb9ed0672b994d25298c82c82babc53adaefe43465b234f4f5003: Status 404 returned error can't find the container with id d6366bdde5feb9ed0672b994d25298c82c82babc53adaefe43465b234f4f5003
Oct 31 00:00:00.340326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:00.340285    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:00:00.340428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:00.340376    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:00:00.340428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:00.340415    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:00:00.991006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:00.990970    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" event={"ID":"bfd37248-4e95-4847-a9b7-07d94e99aa4a","Type":"ContainerStarted","Data":"1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64"}
Oct 31 00:00:00.991379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:00.991024    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" event={"ID":"bfd37248-4e95-4847-a9b7-07d94e99aa4a","Type":"ContainerStarted","Data":"d6366bdde5feb9ed0672b994d25298c82c82babc53adaefe43465b234f4f5003"}
Oct 31 00:00:01.203551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:01.203491    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podStartSLOduration=2.203474286 podStartE2EDuration="2.203474286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.154141132 +0000 UTC m=+1106.444689453" watchObservedRunningTime="2026-10-31 00:00:01.203474286 +0000 UTC m=+1106.494022603"
Oct 31 00:00:01.902854 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:01.902826    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:00:01.904017 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:01.903998    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:00:01.904077 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:01.904042    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 00:00:02.903475 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:02.903438    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:00:02.903918 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:02.903492    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 00:00:03.523805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.523770    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"]
Oct 31 00:00:03.550656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.545836    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.584567 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.584311    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"]
Oct 31 00:00:03.621521 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.621485    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.621742 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.621547    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.621818 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.621744    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.621875 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.621854    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.621965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.621944    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.622056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.622041    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9dh\" (UniqueName: \"kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.627432 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.627410    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"distinct-2-w4-34f1ce-kserve-self-signed-certs\""
Oct 31 00:00:03.723339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723303    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9dh\" (UniqueName: \"kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.723506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723362    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.723506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723405    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.723506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723492    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.723684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723561    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.723684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723616    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.723928 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.723895    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.724062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.724004    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.724062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.724037    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.726116 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.726090    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.726297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.726277    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.782629 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.782553    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9dh\" (UniqueName: \"kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh\") pod \"distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.874183 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.874140    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:03.903748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.903711    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:00:03.904129 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:03.903773    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 00:00:04.276807 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:04.276774    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"]
Oct 31 00:00:04.278358 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:00:04.278330    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0594d4aa_17b2_4daa_b15e_1448f224b9c3.slice/crio-f4fd34c954b098db5c5c570ff780fb7a11a8eba652f906b4c5228924d02f07c4 WatchSource:0}: Error finding container f4fd34c954b098db5c5c570ff780fb7a11a8eba652f906b4c5228924d02f07c4: Status 404 returned error can't find the container with id f4fd34c954b098db5c5c570ff780fb7a11a8eba652f906b4c5228924d02f07c4
Oct 31 00:00:04.903345 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:04.903303    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:00:04.903520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:04.903375    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 00:00:05.017733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:05.017698    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" event={"ID":"0594d4aa-17b2-4daa-b15e-1448f224b9c3","Type":"ContainerStarted","Data":"2026dfe6ded51d0d2ee6bd741a2007a320ef85c6a74d451a617523f6f96a9799"}
Oct 31 00:00:05.018095 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:05.017744    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" event={"ID":"0594d4aa-17b2-4daa-b15e-1448f224b9c3","Type":"ContainerStarted","Data":"f4fd34c954b098db5c5c570ff780fb7a11a8eba652f906b4c5228924d02f07c4"}
Oct 31 00:00:05.133666 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:05.133227    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podStartSLOduration=2.13320847 podStartE2EDuration="2.13320847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:05.133038337 +0000 UTC m=+1110.423586667" watchObservedRunningTime="2026-10-31 00:00:05.13320847 +0000 UTC m=+1110.423756789"
Oct 31 00:00:05.874984 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:05.874953    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:05.903647 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:05.903601    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:00:05.903789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:05.903700    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 00:00:06.903179 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:06.903141    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:00:06.903694 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:06.903225    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 00:00:07.907214 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:07.907184    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:00:07.907705 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:07.907594    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:00:07.908482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:07.908461    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:00:08.324262 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:00:08.324177    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:00:13.874313 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:13.874268    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:13.887183 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:13.887158    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:14.077512 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:00:14.077479    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:00:38.317796 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:00:38.317758    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:01:01.344322 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.344285    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"]
Oct 31 00:01:01.348909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.348888    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.399449 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.399418    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"openshift-service-ca.crt\""
Oct 31 00:01:01.400145 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.399478    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"kube-root-ca.crt\""
Oct 31 00:01:01.400145 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.399606    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"default-dockercfg-dhj9d\""
Oct 31 00:01:01.400301 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.399727    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"distinct-2-w1-5c3374-kserve-self-signed-certs\""
Oct 31 00:01:01.403526 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.403479    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"]
Oct 31 00:01:01.423248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.423216    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.423407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.423257    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.423407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.423280    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.423482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.423427    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.423482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.423463    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrnt\" (UniqueName: \"kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.423561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.423549    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525435 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525385    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525435 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525435    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525457    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525509    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525536    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrnt\" (UniqueName: \"kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525591    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525943 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525918    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.525943 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.525932    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.526063 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.526033    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.527976 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.527952    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.528322 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.528303    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.584056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.584024    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrnt\" (UniqueName: \"kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt\") pod \"distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:01.661396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:01.661301    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:02.035552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:02.035519    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"]
Oct 31 00:01:02.036391 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:01:02.036362    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda685811a_933a_446e_bbbc_b9433d89b356.slice/crio-4e29f60e1caf2ee2b2704dffb7babbb2e546f94a2ada4fa72719781cc9f60486 WatchSource:0}: Error finding container 4e29f60e1caf2ee2b2704dffb7babbb2e546f94a2ada4fa72719781cc9f60486: Status 404 returned error can't find the container with id 4e29f60e1caf2ee2b2704dffb7babbb2e546f94a2ada4fa72719781cc9f60486
Oct 31 00:01:02.337881 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:02.337848    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" event={"ID":"a685811a-933a-446e-bbbc-b9433d89b356","Type":"ContainerStarted","Data":"1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e"}
Oct 31 00:01:02.338074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:02.337892    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" event={"ID":"a685811a-933a-446e-bbbc-b9433d89b356","Type":"ContainerStarted","Data":"4e29f60e1caf2ee2b2704dffb7babbb2e546f94a2ada4fa72719781cc9f60486"}
Oct 31 00:01:02.451065 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:02.451005    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podStartSLOduration=1.450990369 podStartE2EDuration="1.450990369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:02.448380453 +0000 UTC m=+1167.738928772" watchObservedRunningTime="2026-10-31 00:01:02.450990369 +0000 UTC m=+1167.741538686"
Oct 31 00:01:02.661970 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:02.661881    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:08.320903 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:01:08.320849    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:01:11.662499 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:11.662460    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:11.677420 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:11.677392    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:12.405686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:12.405653    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:01:29.122344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.122310    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"]
Oct 31 00:01:29.126486 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.126459    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184339    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184553 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184401    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg4w\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184553 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184488    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184553 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184527    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184629    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184760 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184720    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184784    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184882 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184868    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.184952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.184939    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.227390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.227356    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"]
Oct 31 00:01:29.286164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286129    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg4w\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286172    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286202    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286235    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286281    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286339    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286674 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286412    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286674 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286491    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286674 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286563    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286699    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286804    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.286964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.286923    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.287356 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.287331    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.287465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.287447    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.288568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.288549    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.290218 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.290180    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.345863 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.345820    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.346019 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.345981    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg4w\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.440540 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.440458    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:29.866284 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.866255    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"]
Oct 31 00:01:29.867469 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:01:29.867438    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5ed964_d641_484e_90c3_2830a086604c.slice/crio-0c1c1223991abd932b674525d00b765e07ec59c9d4be2deeb42687d37c35f578 WatchSource:0}: Error finding container 0c1c1223991abd932b674525d00b765e07ec59c9d4be2deeb42687d37c35f578: Status 404 returned error can't find the container with id 0c1c1223991abd932b674525d00b765e07ec59c9d4be2deeb42687d37c35f578
Oct 31 00:01:29.869894 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.869865    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:01:29.869978 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.869927    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:01:29.869978 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:29.869958    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:01:30.517260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:30.517221    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" event={"ID":"4e5ed964-d641-484e-90c3-2830a086604c","Type":"ContainerStarted","Data":"8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99"}
Oct 31 00:01:30.517751 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:30.517279    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" event={"ID":"4e5ed964-d641-484e-90c3-2830a086604c","Type":"ContainerStarted","Data":"0c1c1223991abd932b674525d00b765e07ec59c9d4be2deeb42687d37c35f578"}
Oct 31 00:01:30.682520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:30.682464    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podStartSLOduration=1.68245081 podStartE2EDuration="1.68245081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:30.678846152 +0000 UTC m=+1195.969394483" watchObservedRunningTime="2026-10-31 00:01:30.68245081 +0000 UTC m=+1195.972999127"
Oct 31 00:01:31.449395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:31.449006    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:31.450874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:31.450846    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 00:01:31.451104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:31.451079    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 00:01:32.446156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:32.446116    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 00:01:32.446828 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:32.446793    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 00:01:33.448132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:33.448089    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 00:01:33.448875 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:33.448837    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 00:01:34.452660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:34.449732    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 00:01:34.452660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:34.449790    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 00:01:35.450656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:35.449511    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 00:01:35.450656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:35.449579    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 00:01:36.441473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:36.441430    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 00:01:36.442087 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:36.441494    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 00:01:37.445176 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:37.445142    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:37.445603 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:37.445546    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:37.446182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:37.446158    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:01:38.319071 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:01:38.319027    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:01:45.486394 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.486361    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"]
Oct 31 00:01:45.491730 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.491705    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.590834 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.590798    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"]
Oct 31 00:01:45.641362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641326    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641361    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641619 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641385    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641626    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641799 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641783    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641863 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641847    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2lpj\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641928    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.641982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.641972    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.642064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.642052    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.743693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.743591    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.743693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.743667    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.743930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.743711    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.743930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.743760    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.743930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.743855    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.743930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.743910    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744046    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744061    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744115    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744195    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744262    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2lpj\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744542 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744519    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.744731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.744708    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.745552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.745528    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.746072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.746054    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.746415 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.746398    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.802202 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.802166    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.802459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.802436    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2lpj\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:45.805207 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:45.805181    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:46.237649 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.237595    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"]
Oct 31 00:01:46.264779 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:01:46.264748    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f65b7e_e40c_4a56_afcc_f23f00778bf6.slice/crio-031831003ddcde98775e122993d671d40c230b43ad634b9ec4bd305c5bbd60da WatchSource:0}: Error finding container 031831003ddcde98775e122993d671d40c230b43ad634b9ec4bd305c5bbd60da: Status 404 returned error can't find the container with id 031831003ddcde98775e122993d671d40c230b43ad634b9ec4bd305c5bbd60da
Oct 31 00:01:46.267257 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.267221    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:01:46.267364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.267295    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:01:46.267364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.267324    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:01:46.636824 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.636784    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" event={"ID":"34f65b7e-e40c-4a56-afcc-f23f00778bf6","Type":"ContainerStarted","Data":"975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef"}
Oct 31 00:01:46.637195 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.636849    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" event={"ID":"34f65b7e-e40c-4a56-afcc-f23f00778bf6","Type":"ContainerStarted","Data":"031831003ddcde98775e122993d671d40c230b43ad634b9ec4bd305c5bbd60da"}
Oct 31 00:01:46.806167 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:46.806129    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:47.806963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:47.806921    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:01:47.807428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:47.807002    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 00:01:48.806410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:48.806371    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:01:48.806593 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:48.806449    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 00:01:49.805485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:49.805457    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:01:49.805947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:49.805506    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 00:01:50.805937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:50.805900    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:01:50.806336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:50.805959    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 00:01:51.806507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:51.806475    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:01:51.807004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:51.806527    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 00:01:53.176227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:53.176193    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:53.176805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:53.176780    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:53.177578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:53.177553    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:01:53.337776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:01:53.337725    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podStartSLOduration=8.337712175 podStartE2EDuration="8.337712175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:46.759502379 +0000 UTC m=+1212.050050698" watchObservedRunningTime="2026-10-31 00:01:53.337712175 +0000 UTC m=+1218.628260493"
Oct 31 00:02:00.579905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.579870    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"]
Oct 31 00:02:00.584069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.584050    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.685066 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.685029    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"]
Oct 31 00:02:00.742747 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.742714    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.742920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.742827    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tzt2\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.742920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.742915    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.743007 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.742956    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.743057 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.743037    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.743093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.743066    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.743130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.743101    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.743211 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.743195    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.743248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.743223    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844359    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844401    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844605 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844494    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844605 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844548    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzt2\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844749 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844623    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844749 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844720    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844858 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844773    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844858 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844798    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.844858 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844821    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.845007 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.844850    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.845097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.845074    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.845231 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.845207    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.845290 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.845212    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.845290 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.845255    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.846907 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.846882    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.847098 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.847082    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.902852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.902818    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:00.902998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:00.902937    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzt2\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:01.197755 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.197670    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:01.620297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.620263    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"]
Oct 31 00:02:01.621720 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:02:01.621686    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6518a1b2_9688_417d_a45d_3ac2e9b98967.slice/crio-47da4a63cfd8bfc72250abbd1a871ecf8452ea4de1c6f8c3931fc79008a882cf WatchSource:0}: Error finding container 47da4a63cfd8bfc72250abbd1a871ecf8452ea4de1c6f8c3931fc79008a882cf: Status 404 returned error can't find the container with id 47da4a63cfd8bfc72250abbd1a871ecf8452ea4de1c6f8c3931fc79008a882cf
Oct 31 00:02:01.624929 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.624894    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:01.625040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.624972    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:01.625040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.625014    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:01.742243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.742198    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" event={"ID":"6518a1b2-9688-417d-a45d-3ac2e9b98967","Type":"ContainerStarted","Data":"5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0"}
Oct 31 00:02:01.742423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.742269    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" event={"ID":"6518a1b2-9688-417d-a45d-3ac2e9b98967","Type":"ContainerStarted","Data":"47da4a63cfd8bfc72250abbd1a871ecf8452ea4de1c6f8c3931fc79008a882cf"}
Oct 31 00:02:01.874694 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:01.874563    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podStartSLOduration=1.874543648 podStartE2EDuration="1.874543648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:01.871440391 +0000 UTC m=+1227.161988710" watchObservedRunningTime="2026-10-31 00:02:01.874543648 +0000 UTC m=+1227.165091968"
Oct 31 00:02:02.198341 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:02.198249    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:02.200168 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:02.200142    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:02.200237 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:02.200192    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:03.198674 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:03.198591    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:03.199295 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:03.199240    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:04.201177 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:04.201139    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:04.201825 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:04.201780    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:04.346781 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:04.346730    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 00:02:04.388807 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:04.388758    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:04.478716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:04.477723    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:04.974658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:04.973782    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.017659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.016960    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.071659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.070872    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.091874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.091828    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.200658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.200595    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:05.200950 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.200923    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:05.361669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.359757    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.361669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.360450    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:02:05.389394 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.389353    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:02:05.390050 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.390029    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:02:05.401772 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.401732    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.485782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.485731    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.486503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.486471    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:02:05.561481 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.518213    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.600543 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.600515    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:02:05.737843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.737744    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.839329 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.822034    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.981362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.981314    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:05.984591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:05.984483    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:02:06.022477 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:06.022392    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:02:06.025066 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:06.025043    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:02:06.025748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:06.025665    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:02:06.198093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:06.198057    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:06.198244 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:06.198123    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:07.198717 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:07.198682    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:07.199106 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:07.198744    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:08.198361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:08.198317    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:02:08.198562 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:08.198375    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 00:02:08.319995 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:02:08.319957    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:02:09.251116 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:09.251081    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:09.251445 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:09.251423    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:09.252370 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:09.252345    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:02:29.946951 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:29.946917    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"]
Oct 31 00:02:29.950920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:29.950897    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019042    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019277 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019163    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019277 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019241    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019277 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019268    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019452    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5nwl\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019531 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019492    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019531 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019519    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019598    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.019730 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.019707    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.121659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.121606    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.121853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.121666    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.121853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.121690    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.121853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.121732    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.121853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.121849    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.121928    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122016    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122049    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122069    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122135    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5nwl\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122135    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122368 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122347    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122422    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.122533 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.122513    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.124253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.124231    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.124511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.124493    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.149847 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.149814    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"]
Oct 31 00:02:30.179926 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.179891    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.180119 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.180009    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5nwl\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.264336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.264253    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:30.688347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.688320    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"]
Oct 31 00:02:30.689400 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:02:30.689372    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42004e9_c4d2_435a_8f4b_0c283473b484.slice/crio-f7435768a5d7ca8f738e62e7fd751d8d6c3391f76ae1f9f3232009bd6294957b WatchSource:0}: Error finding container f7435768a5d7ca8f738e62e7fd751d8d6c3391f76ae1f9f3232009bd6294957b: Status 404 returned error can't find the container with id f7435768a5d7ca8f738e62e7fd751d8d6c3391f76ae1f9f3232009bd6294957b
Oct 31 00:02:30.692266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.692231    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:30.692351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.692314    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:30.692389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.692350    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:30.845575 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.845541    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"]
Oct 31 00:02:30.850323 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.850301    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.895018 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.894622    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"]
Oct 31 00:02:30.932846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.932812    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.932865    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.932970    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933175 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.933032    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzcw\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933233 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.933188    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933287 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.933242    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933399 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.933365    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.933560    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:30.933687 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:30.933596    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042153    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042218    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042263    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042289    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzcw\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042394    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042421    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042528    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042664    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.042974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.042694    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.043264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.043240    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.044004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.043978    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.045077 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.044844    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.046440 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.046279    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" event={"ID":"e42004e9-c4d2-435a-8f4b-0c283473b484","Type":"ContainerStarted","Data":"603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67"}
Oct 31 00:02:31.046440 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.046341    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" event={"ID":"e42004e9-c4d2-435a-8f4b-0c283473b484","Type":"ContainerStarted","Data":"f7435768a5d7ca8f738e62e7fd751d8d6c3391f76ae1f9f3232009bd6294957b"}
Oct 31 00:02:31.046975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.046949    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.047896 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.047875    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.048047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.048033    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.048090 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.048033    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.103912 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.103881    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.104189 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.104165    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzcw\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.168872 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.168837    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:31.169695 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.169616    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podStartSLOduration=2.169551538 podStartE2EDuration="2.169551538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:31.166857585 +0000 UTC m=+1256.457405903" watchObservedRunningTime="2026-10-31 00:02:31.169551538 +0000 UTC m=+1256.460099856"
Oct 31 00:02:31.264522 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.264487    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:31.266053 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.266028    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:31.266201 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.266075    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:31.603056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.603014    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"]
Oct 31 00:02:31.604004 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:02:31.603969    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6061f4cc_491a_493d_ab55_1836ab0b675a.slice/crio-9e79d38d815677529d518e0d2faa7644b56c7995e080d49158a6a9b678c844a2 WatchSource:0}: Error finding container 9e79d38d815677529d518e0d2faa7644b56c7995e080d49158a6a9b678c844a2: Status 404 returned error can't find the container with id 9e79d38d815677529d518e0d2faa7644b56c7995e080d49158a6a9b678c844a2
Oct 31 00:02:31.606935 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.606902    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:31.607030 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.606966    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:31.607030 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:31.606993    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:02:32.066784 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.066688    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" event={"ID":"6061f4cc-491a-493d-ab55-1836ab0b675a","Type":"ContainerStarted","Data":"68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611"}
Oct 31 00:02:32.067230 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.066796    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" event={"ID":"6061f4cc-491a-493d-ab55-1836ab0b675a","Type":"ContainerStarted","Data":"9e79d38d815677529d518e0d2faa7644b56c7995e080d49158a6a9b678c844a2"}
Oct 31 00:02:32.169128 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.169050    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:32.170614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.170589    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:32.170762 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.170651    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:32.182750 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.182697    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podStartSLOduration=2.182684284 podStartE2EDuration="2.182684284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:32.180807937 +0000 UTC m=+1257.471356254" watchObservedRunningTime="2026-10-31 00:02:32.182684284 +0000 UTC m=+1257.473232802"
Oct 31 00:02:32.265561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.265526    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:32.265758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:32.265604    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:33.178101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:33.178063    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:33.178627 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:33.178159    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:33.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:33.275069    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:33.275445 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:33.275416    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:34.176116 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:34.176076    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:34.177192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:34.177159    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:34.277805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:34.277748    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:34.278315 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:34.277818    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:35.177688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:35.177152    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:35.177688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:35.177262    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:35.268128 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:35.268087    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:35.268446 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:35.268417    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:36.179660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:36.172141    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:36.179660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:36.172223    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:36.268076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:36.268037    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:36.268377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:36.268349    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:37.170178 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:37.170140    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:37.170385 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:37.170210    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:37.265373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:37.265334    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:02:37.265811 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:37.265407    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 00:02:38.169582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:38.169539    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:02:38.169800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:38.169649    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 00:02:38.324944 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:02:38.324905    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:02:38.391215 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:38.391169    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:38.391958 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:38.391920    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:38.392799 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:38.392773    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:02:40.229818 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:40.229779    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:40.231139 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:40.230398    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:40.881546 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:40.881495    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.009780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.009734    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.058388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.057222    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.058700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.058670    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.076585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.076535    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.079012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.078976    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.103978 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.103877    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.293721 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.293029    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:41.293721 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.293098    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.412658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.405746    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.412658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.406577    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.608795 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.608744    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.742978 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.740724    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.765705 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.764747    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.857657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:41.857428    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.037748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.011877    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.037748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.012783    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:02:42.076145 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.076099    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.094937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.094890    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.095583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.095552    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.118126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.101997    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:02:42.118501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.118477    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:02:42.118765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.118747    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:02:42.148660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.120578    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.200069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.142777    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.204659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.200673    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 31 00:02:42.230524 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.230493    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:02:42.392783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.392731    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.411285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.411220    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.411809 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.411779    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:02:42.412240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.412216    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.436969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.436890    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:02:42.616693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.561822    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:42.616693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.561903    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.696974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.696869    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.748664 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.699767    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.748664 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.700161    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:02:42.802661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.801735    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.821669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.820724    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 31 00:02:42.906660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.890075    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.963717 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:42.939051    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.108658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.072651    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.133730 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.133672    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.276719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.213696    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.276719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.213775    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.276719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.213816    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.276719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.094864    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.415677 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.415605    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.428359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.420503    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.511670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.510308    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:02:43.511670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.510380    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:02:43.611658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.610021    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:43.611658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.610086    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.806620 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.806577    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.807869 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.807831    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.872936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:43.872847    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:02:44.090965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.090924    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 00:02:44.100781 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.093107    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:44.106379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.106063    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:44.113600 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.113575    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:44.118476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.118450    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:02:44.119726 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.119702    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:02:44.122390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.122166    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:02:44.148176 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.148089    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:02:44.214756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.214709    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:44.215325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.215290    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 00:02:44.241900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.241870    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:44.242476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.242449    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:02:44.243535 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.241709    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:44.255275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.255247    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:02:44.259740 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.256329    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:02:44.261682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.261664    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 31 00:02:44.263914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.263896    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:02:44.268197 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.268180    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:02:44.270042 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.269410    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:02:44.275070 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:44.275055    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 31 00:02:59.527558 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.527520    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"]
Oct 31 00:02:59.531712 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.531690    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.633763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.633717    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"]
Oct 31 00:02:59.664045 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664009    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664185    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664215    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664244    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664284    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664401 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664373    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664436 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664403    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664462    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55bd\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.664585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.664572    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.765937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.765894    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.765947    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.765988    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766013    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766062    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766099    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766153    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55bd\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766209    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766316    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766436    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766744 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766710    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.766947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766925    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.767013 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.766980    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.767075 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.767004    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.768883 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.768861    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.769126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.769107    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.824977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.824899    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.825160 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.825135    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55bd\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:02:59.845437 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:02:59.845408    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:03:00.280994 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:00.280866    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"]
Oct 31 00:03:00.284220 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:03:00.284174    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0cab75_97d3_444b_91cf_a53286aa4ccc.slice/crio-02f3e1e7b7b1bdf80d6e37d7b05b1b95ec5fa20d48c9cd1d0cd52ec68265f7b9 WatchSource:0}: Error finding container 02f3e1e7b7b1bdf80d6e37d7b05b1b95ec5fa20d48c9cd1d0cd52ec68265f7b9: Status 404 returned error can't find the container with id 02f3e1e7b7b1bdf80d6e37d7b05b1b95ec5fa20d48c9cd1d0cd52ec68265f7b9
Oct 31 00:03:00.286529 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:00.286494    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:00.286657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:00.286573    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:00.286657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:00.286620    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:01.079759 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.079721    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"]
Oct 31 00:03:01.080225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.079947    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy" containerID="cri-o://22c3bc404b6e6b7eaf3b658f6c8c2de0ef57ed37d114b42be716c42eacdd9858" gracePeriod=30
Oct 31 00:03:01.260379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.260340    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" event={"ID":"5f0cab75-97d3-444b-91cf-a53286aa4ccc","Type":"ContainerStarted","Data":"21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800"}
Oct 31 00:03:01.260594 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.260439    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" event={"ID":"5f0cab75-97d3-444b-91cf-a53286aa4ccc","Type":"ContainerStarted","Data":"02f3e1e7b7b1bdf80d6e37d7b05b1b95ec5fa20d48c9cd1d0cd52ec68265f7b9"}
Oct 31 00:03:01.377704 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.377649    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podStartSLOduration=2.377619325 podStartE2EDuration="2.377619325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:01.374203465 +0000 UTC m=+1286.664751785" watchObservedRunningTime="2026-10-31 00:03:01.377619325 +0000 UTC m=+1286.668167642"
Oct 31 00:03:01.846975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.846941    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:03:01.850767 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.850733    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:03:01.851010 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:01.850976    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:03:02.853147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:02.853106    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:03:02.853822 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:02.853188    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:03:03.847563 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:03.847044    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:03:03.847563 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:03.847113    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:03:04.854127 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:04.854083    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:03:04.854560 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:04.854148    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:03:05.848658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:05.848029    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:03:05.848658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:05.848095    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:03:06.298707 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.298670    4466 generic.go:356] "Generic (PLEG): container finished" podID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerID="22c3bc404b6e6b7eaf3b658f6c8c2de0ef57ed37d114b42be716c42eacdd9858" exitCode=0
Oct 31 00:03:06.298707 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.298687    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" event={"ID":"39910e80-d7fb-488b-a65b-246e3f74bc90","Type":"ContainerDied","Data":"22c3bc404b6e6b7eaf3b658f6c8c2de0ef57ed37d114b42be716c42eacdd9858"}
Oct 31 00:03:06.373345 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.373284    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 31 00:03:06.495071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495040    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495086    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495123    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495266    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495285    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495309    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495492 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495362    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495492 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495418    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495492 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495414    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.495656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495544    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket\") pod \"39910e80-d7fb-488b-a65b-246e3f74bc90\" (UID: \"39910e80-d7fb-488b-a65b-246e3f74bc90\") "
Oct 31 00:03:06.495724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495669    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.495991 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495918    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.495991 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495946    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:06.495991 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.495968    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.496453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.496428    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.496453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.496453    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.496656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.496466    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.496656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.496480    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39910e80-d7fb-488b-a65b-246e3f74bc90-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.496656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.496493    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.497542 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.497516    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:03:06.497679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.497654    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.497836 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.497808    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.498252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.498224    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s" pod "39910e80-d7fb-488b-a65b-246e3f74bc90" (UID: "39910e80-d7fb-488b-a65b-246e3f74bc90"). InnerVolumeSpecName "kube-api-access-tvz4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.598794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.598748    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvz4s\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-kube-api-access-tvz4s\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.598794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.598796    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.599003 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.598812    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.599003 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.598826    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39910e80-d7fb-488b-a65b-246e3f74bc90-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.846827 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.846797    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:03:06.846963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:06.846859    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 00:03:07.308325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.308227    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"
Oct 31 00:03:07.308325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.308255    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg" event={"ID":"39910e80-d7fb-488b-a65b-246e3f74bc90","Type":"ContainerDied","Data":"e6dade7eac739147438300ac299d7416b8fc04293336570ac6f3ae80635714cb"}
Oct 31 00:03:07.308325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.308323    4466 scope.go:122] "RemoveContainer" containerID="22c3bc404b6e6b7eaf3b658f6c8c2de0ef57ed37d114b42be716c42eacdd9858"
Oct 31 00:03:07.513596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.513558    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"]
Oct 31 00:03:07.559355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.559253    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-vt4mg"]
Oct 31 00:03:07.850840 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.850811    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:03:07.851091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.851072    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:03:07.851844 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:07.851828    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:03:08.319056 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:03:08.318930    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:03:08.828021 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:08.827984    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" path="/var/lib/kubelet/pods/39910e80-d7fb-488b-a65b-246e3f74bc90/volumes"
Oct 31 00:03:20.915273 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:20.915040    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:20.938054 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:20.938012    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 00:03:21.013195 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.012784    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.107254 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.107199    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.119688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.107907    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.210720 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.209895    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.230267 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.230207    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.261658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.260830    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.261658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.261231    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:03:21.261658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.261503    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 31 00:03:21.265656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.262811    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.439621 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.439578    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.440747 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.440716    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.944156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.944104    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.945666 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:21.945613    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:03:22.013782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.013741    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.028657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.015294    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:03:22.108793 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.108750    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.152672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.126766    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:03:22.152672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.127852    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.152672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.141696    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:03:22.269826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.269734    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.279528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.270617    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:03:22.279528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.272400    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:03:22.279528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.274135    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:03:22.279528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.274187    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:03:22.279528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.274236    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 31 00:03:22.289657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.285356    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 31 00:03:22.289657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.287767    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 31 00:03:22.322656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.302011    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.337926 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.314119    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 00:03:22.339276 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.339249    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:03:22.339485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.339470    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:03:22.376071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.376028    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:03:22.376275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.376097    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.387769 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.387726    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.443269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.442041    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:22.444713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.443565    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:03:22.478857 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.458130    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 00:03:22.478857 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.458785    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:03:22.695658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:22.691705    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.061210 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.061104    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.204663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.071891    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.249659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.139192    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.309668 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.289907    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.328757 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.313760    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.368152 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.278497    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.406794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.406745    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.429568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.429516    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.460274 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.460219    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.699428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.699389    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg" podUID="d6c3372c-3895-4d31-a94a-b7512c346169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.699980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.699840    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.710670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.694816    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.770033 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.768034    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.926012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.925976    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:03:23.926303 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.926285    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:03:23.926523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.926507    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 31 00:03:23.930732 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.930708    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:03:23.930917 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.930905    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:03:23.931033 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.931023    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:03:23.931136 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.931127    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:03:23.931452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.931434    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:03:23.949930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.949826    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:03:23.953782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:23.953755    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 31 00:03:24.115769 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:24.115050    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:03:24.153754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:24.153495    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:03:38.322487 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:03:38.322440    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:03:42.738451 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.738419    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"]
Oct 31 00:03:42.739012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.738999    4466 cpu_manager.go:414] "containerMap: removing container" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy"
Oct 31 00:03:42.739055 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.739013    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy"
Oct 31 00:03:42.739143 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.739134    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="39910e80-d7fb-488b-a65b-246e3f74bc90" containerName="istio-proxy"
Oct 31 00:03:42.744064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.744041    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.793079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.793047    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 00:03:42.836081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836047    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.836272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836208    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmww\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.836336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836304    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.836384 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836337    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.836384 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836352    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.836523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836502    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.836558 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.836547    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.845174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.845144    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"]
Oct 31 00:03:42.894071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.894034    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 00:03:42.894290 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.894095    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 00:03:42.894290 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.894170    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbdd5\""
Oct 31 00:03:42.938504 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.938467    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.938741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.938539    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.938741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.938568    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.938741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.938692    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.938931 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.938742    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.938931 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.938892    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.939045 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.939028    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmww\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.939317 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.939284    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.941181 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.941156    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.941310 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.941296    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.941539 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.941521    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.941578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.941548    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.997038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.996968    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:42.997164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:42.997119    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmww\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wqjzk\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:43.056833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:43.056792    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:43.479746 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:43.479711    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"]
Oct 31 00:03:43.511776 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:03:43.511748    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71235c6f_1799_4f13_8eab_61b18c47a390.slice/crio-cdd95c92cd3886c31b9a0b9a577ecfcf78709989513d38831958df2ce9e28ec4 WatchSource:0}: Error finding container cdd95c92cd3886c31b9a0b9a577ecfcf78709989513d38831958df2ce9e28ec4: Status 404 returned error can't find the container with id cdd95c92cd3886c31b9a0b9a577ecfcf78709989513d38831958df2ce9e28ec4
Oct 31 00:03:43.514315 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:43.514280    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:43.514444 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:43.514356    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:44.128247 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.128210    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" event={"ID":"71235c6f-1799-4f13-8eab-61b18c47a390","Type":"ContainerStarted","Data":"d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc"}
Oct 31 00:03:44.128742 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.128273    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" event={"ID":"71235c6f-1799-4f13-8eab-61b18c47a390","Type":"ContainerStarted","Data":"cdd95c92cd3886c31b9a0b9a577ecfcf78709989513d38831958df2ce9e28ec4"}
Oct 31 00:03:44.128742 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.128299    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:44.130192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.130163    4466 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wqjzk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:03:44.130306 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.130210    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:03:44.243789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.243732    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" podStartSLOduration=2.243716987 podStartE2EDuration="2.243716987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:44.242529287 +0000 UTC m=+1329.533077606" watchObservedRunningTime="2026-10-31 00:03:44.243716987 +0000 UTC m=+1329.534265305"
Oct 31 00:03:44.937524 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.937478    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"]
Oct 31 00:03:44.942264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:44.942238    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.047753 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.047690    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"]
Oct 31 00:03:45.072785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.072739    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.072982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.072804    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.072982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.072844    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.072982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.072943    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.072982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.072970    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.073208 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.073012    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.073208 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.073064    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.073524 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.073496    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkrk\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.073610 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.073574    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.135108 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.135076    4466 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wqjzk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:03:45.135505 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.135137    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:03:45.175127 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175086    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175147    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175318    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkrk\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175433 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175350    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175432    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175464    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175601 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175501    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175593    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175704    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175725    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175827 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175792    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.175964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.175946    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.176049 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.176024    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.176350 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.176331    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.177959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.177929    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.178097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.178078    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.225806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.225714    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"]
Oct 31 00:03:45.230717 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.230691    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.246394 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.246360    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.246925 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.246900    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkrk\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.262119 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.262090    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:45.333251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.332389    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"]
Oct 31 00:03:45.379742 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.379704    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.379866 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.379773    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbs2h\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.379866 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.379853    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.379948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.379906    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.379997 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.379981    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.380113 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.380099    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.380150 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.380126    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.380217 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.380203    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.380317 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.380304    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481394    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481628 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481485    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481628 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481514    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbs2h\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481628 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481554    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481628 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481598    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481675    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481769    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481842 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481788    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481792    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481857    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.481965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481946    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.482017 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.481987    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.482179 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.482148    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.482457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.482434    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.484017 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.483999    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.484333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.484317    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.546108 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.546071    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.546268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.546144    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbs2h\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:45.744109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.744084    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"]
Oct 31 00:03:45.744565 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:03:45.744539    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a87bf5_30da_4901_8be0_e399f44cf476.slice/crio-c13d72d86845f87d0f731ef1cdb869805fce142fc577c0cd7eac5b4afa157653 WatchSource:0}: Error finding container c13d72d86845f87d0f731ef1cdb869805fce142fc577c0cd7eac5b4afa157653: Status 404 returned error can't find the container with id c13d72d86845f87d0f731ef1cdb869805fce142fc577c0cd7eac5b4afa157653
Oct 31 00:03:45.747363 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.747327    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:45.747450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.747394    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:45.747450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.747426    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:45.846086 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:45.846051    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:46.143648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.143603    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" event={"ID":"e6a87bf5-30da-4901-8be0-e399f44cf476","Type":"ContainerStarted","Data":"6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c"}
Oct 31 00:03:46.144030 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.143689    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" event={"ID":"e6a87bf5-30da-4901-8be0-e399f44cf476","Type":"ContainerStarted","Data":"c13d72d86845f87d0f731ef1cdb869805fce142fc577c0cd7eac5b4afa157653"}
Oct 31 00:03:46.262447 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.262383    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podStartSLOduration=2.262367162 podStartE2EDuration="2.262367162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:46.260363631 +0000 UTC m=+1331.550911950" watchObservedRunningTime="2026-10-31 00:03:46.262367162 +0000 UTC m=+1331.552915481"
Oct 31 00:03:46.262447 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.262434    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:46.264582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.264551    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:46.264735 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.264598    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:46.290626 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.286926    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"]
Oct 31 00:03:46.297911 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:03:46.297862    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece29ed1_b432_4c6b_9156_031a2f567516.slice/crio-f8ddff1f06135a82535c5c7123e36e5440c09a2a09e2aa1703e60d69e8d763e3 WatchSource:0}: Error finding container f8ddff1f06135a82535c5c7123e36e5440c09a2a09e2aa1703e60d69e8d763e3: Status 404 returned error can't find the container with id f8ddff1f06135a82535c5c7123e36e5440c09a2a09e2aa1703e60d69e8d763e3
Oct 31 00:03:46.301873 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.301835    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:46.302085 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.302057    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:46.302200 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:46.302111    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:03:47.185043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.185001    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" event={"ID":"ece29ed1-b432-4c6b-9156-031a2f567516","Type":"ContainerStarted","Data":"f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c"}
Oct 31 00:03:47.185426 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.185064    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" event={"ID":"ece29ed1-b432-4c6b-9156-031a2f567516","Type":"ContainerStarted","Data":"f8ddff1f06135a82535c5c7123e36e5440c09a2a09e2aa1703e60d69e8d763e3"}
Oct 31 00:03:47.277091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.269772    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:47.277091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.269853    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:47.327178 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.327123    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podStartSLOduration=2.3271084699999998 podStartE2EDuration="2.32710847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:47.32558048 +0000 UTC m=+1332.616128799" watchObservedRunningTime="2026-10-31 00:03:47.32710847 +0000 UTC m=+1332.617656782"
Oct 31 00:03:47.871797 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.871758    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:47.873581 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.873553    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:03:47.873854 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:47.873827    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:03:48.165717 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:48.164019    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:03:48.269657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:48.267222    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:48.269657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:48.267280    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:48.862656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:48.860040    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:03:48.862656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:48.860117    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:03:49.268603 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:49.268509    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:49.268912 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:49.268881    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:49.866485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:49.861992    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:03:49.866485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:49.862068    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:03:50.285199 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:50.285103    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:50.285377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:50.285189    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:50.864659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:50.864049    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:03:50.864659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:50.864140    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:03:51.275716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:51.275323    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:51.275716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:51.275399    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:51.862766 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:51.853876    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:03:51.862766 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:51.861789    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:03:52.262573 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:52.262483    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:03:52.262573 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:52.262538    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 00:03:52.848691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:52.846921    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 00:03:52.848691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:52.846994    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 00:03:53.267622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:53.267544    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:53.268011 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:53.267989    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:53.268900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:53.268877    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:03:53.871022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:53.870990    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:53.872068 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:53.872045    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:03:53.872838 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:03:53.872818    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:04:02.885683 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:02.858348    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:02.885683 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:02.859240    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 31 00:04:02.922658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:02.922133    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 00:04:03.008696 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:02.982966    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.039689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.038610    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 00:04:03.205719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.205195    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.228361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.228311    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.304577 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.304529    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 31 00:04:03.378299 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.376772    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.880132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.877794    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.880132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.878844    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:04:03.918660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.911528    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.918660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.912017    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 00:04:03.918660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.914202    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:04:03.974799 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.974744    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.990658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.987947    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.990658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:03.988442    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:04:04.041920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.038842    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.041920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.039235    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:04:04.301716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.301298    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.301896 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.301723    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:04:04.333685 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.311708    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w" podUID="beae846e-f30e-48a7-93c0-04fae50b694c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 31 00:04:04.333685 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.313147    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 31 00:04:04.398804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.398752    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 31 00:04:04.408671 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.399487    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:04:04.435355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.434778    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 00:04:04.991694 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:04.971353    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 00:04:05.116663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.116602    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.117784 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.117589    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.124299 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.124260    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.132840 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.132794    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 00:04:05.144666 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.144590    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.191024 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.190982    4466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246034    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246104    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246151    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246187    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246219    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246259    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:04:05.254839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.246288    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:04:05.293668 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.290552    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:04:05.293668 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.293578    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:04:05.346559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.346523    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.359040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.359012    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:04:05.393255 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:05.393228    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:04:08.319185 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:04:08.319146    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:04:31.844206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.844167    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"]
Oct 31 00:04:31.848360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.848335    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923063    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923282 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923189    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6t7l\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923282 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923247    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923282    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923418 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923398    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923460 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923429    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923460    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923526 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923501    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:31.923559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:31.923544    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.025586 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025543    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.025804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025626    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6t7l\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.025804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025673    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.025804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025703    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.025922 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025885    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.025969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025924    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.025981    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026061 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026009    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026061 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026046    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026051    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026092    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026229    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026422 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026399    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.026797 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.026775    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.028229 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.028205    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.028410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.028393    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.048460 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.048426    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"]
Oct 31 00:04:32.084188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.084153    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.084686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.084659    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6t7l\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l\") pod \"maas-default-gateway-openshift-default-8f6f87f59-sgc28\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.163445 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.163361    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:32.608597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.608565    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"]
Oct 31 00:04:32.609977 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:04:32.609951    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d97aa5_d5f6_4263_ac67_5c83b5b742aa.slice/crio-e61dd5d14a7aeafe5a7555d315391837bf66152c13c1ac0ca5148c38223bae91 WatchSource:0}: Error finding container e61dd5d14a7aeafe5a7555d315391837bf66152c13c1ac0ca5148c38223bae91: Status 404 returned error can't find the container with id e61dd5d14a7aeafe5a7555d315391837bf66152c13c1ac0ca5148c38223bae91
Oct 31 00:04:32.612889 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.612856    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:04:32.612969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.612925    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:04:32.612969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:32.612955    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:04:33.385728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:33.385688    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" event={"ID":"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa","Type":"ContainerStarted","Data":"fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333"}
Oct 31 00:04:33.386184 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:33.385755    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" event={"ID":"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa","Type":"ContainerStarted","Data":"e61dd5d14a7aeafe5a7555d315391837bf66152c13c1ac0ca5148c38223bae91"}
Oct 31 00:04:33.555778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:33.555715    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podStartSLOduration=2.555699186 podStartE2EDuration="2.555699186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:33.553376696 +0000 UTC m=+1378.843925015" watchObservedRunningTime="2026-10-31 00:04:33.555699186 +0000 UTC m=+1378.846247568"
Oct 31 00:04:34.164073 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.164033    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:34.165621 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.165594    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:34.165779 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.165660    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:34.260745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.260711    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"]
Oct 31 00:04:34.265758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.265735    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.316424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.315238    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e0751b65-serving-cert\""
Oct 31 00:04:34.320037 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.320007    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"]
Oct 31 00:04:34.372324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.372280    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsdc\" (UniqueName: \"kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.372530 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.372405    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.372530 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.372511    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.372622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.372593    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.473948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.473845    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.473948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.473914    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.474506 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:04:34.474095    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-serving-cert: secret "maas-api-e2e-mt-e0751b65-serving-cert" not found
Oct 31 00:04:34.474506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.474176    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsdc\" (UniqueName: \"kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.474506 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:04:34.474196    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls podName:d7683bc7-a477-45aa-ac74-e0759935f599 nodeName:}" failed. No retries permitted until 2026-10-31 00:04:34.974173969 +0000 UTC m=+1380.264722282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls") pod "maas-api-e2e-mt-e0751b65-7447786c8f-425lj" (UID: "d7683bc7-a477-45aa-ac74-e0759935f599") : secret "maas-api-e2e-mt-e0751b65-serving-cert" not found
Oct 31 00:04:34.474506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.474292    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.474953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.474934    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.476428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.476405    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.546176 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.546143    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsdc\" (UniqueName: \"kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.982380 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.982338    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:34.984968 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:34.984936    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\") pod \"maas-api-e2e-mt-e0751b65-7447786c8f-425lj\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:35.163803 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:35.163767    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:35.164047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:35.163827    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:35.180938 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:35.180894    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:35.584162 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:35.584128    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"]
Oct 31 00:04:35.584506 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:04:35.584159    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7683bc7_a477_45aa_ac74_e0759935f599.slice/crio-7dbf352fbe7ce7e335594d6cfc23400cb2079e421e74866f6a08d64400e27a10 WatchSource:0}: Error finding container 7dbf352fbe7ce7e335594d6cfc23400cb2079e421e74866f6a08d64400e27a10: Status 404 returned error can't find the container with id 7dbf352fbe7ce7e335594d6cfc23400cb2079e421e74866f6a08d64400e27a10
Oct 31 00:04:35.586328 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:35.586310    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:04:36.164503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:36.164470    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:36.164730 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:36.164524    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:36.422693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:36.422603    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj" event={"ID":"d7683bc7-a477-45aa-ac74-e0759935f599","Type":"ContainerStarted","Data":"7dbf352fbe7ce7e335594d6cfc23400cb2079e421e74866f6a08d64400e27a10"}
Oct 31 00:04:37.164029 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:37.163992    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:37.164454 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:37.164054    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:37.464722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:37.464593    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj" event={"ID":"d7683bc7-a477-45aa-ac74-e0759935f599","Type":"ContainerStarted","Data":"c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147"}
Oct 31 00:04:37.465552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:37.465528    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:37.582357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:37.582267    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj" podStartSLOduration=2.705275785 podStartE2EDuration="3.582246967s" totalImagesPullingTime="876.971182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:37.57928487 +0000 UTC m=+1382.869833191" watchObservedRunningTime="2026-10-31 00:04:37.582246967 +0000 UTC m=+1382.872795321"
Oct 31 00:04:38.164275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:38.164232    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:38.164781 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:38.164296    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:38.319515 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:04:38.319476    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:04:39.163728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:39.163689    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:39.163928 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:39.163762    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:40.164424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:40.164375    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-sgc28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 00:04:40.164927 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:40.164458    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 00:04:41.619952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:41.619908    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:41.620530 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:41.620509    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:41.621351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:41.621333    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:04:43.477051 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:43.477021    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:04:53.457831 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.457547    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"]
Oct 31 00:04:53.475039 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.475011    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.522419 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.522379    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"]
Oct 31 00:04:53.586658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.571726    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d32a7cc0-serving-cert\""
Oct 31 00:04:53.788759 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.787550    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8rh\" (UniqueName: \"kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.857672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.836517    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.857672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.836665    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.857672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.836746    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.950829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.940241    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.950829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.940337    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.950829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.940627    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8rh\" (UniqueName: \"kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.950829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.940785    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:53.975547 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:53.952363    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:54.047697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:54.047475    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:54.048656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:54.048617    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:54.098735 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:54.098545    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8rh\" (UniqueName: \"kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh\") pod \"maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:54.111942 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:54.110151    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:54.531920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:54.530953    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"]
Oct 31 00:04:54.534042 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:04:54.534002    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6639b9bc_4d27_472f_91d9_be13bab2be96.slice/crio-7a2015d46b4ac83500984abb62e7e26a4338988f1c15b2003e25b29b85d86c2c WatchSource:0}: Error finding container 7a2015d46b4ac83500984abb62e7e26a4338988f1c15b2003e25b29b85d86c2c: Status 404 returned error can't find the container with id 7a2015d46b4ac83500984abb62e7e26a4338988f1c15b2003e25b29b85d86c2c
Oct 31 00:04:54.746307 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:54.746265    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v" event={"ID":"6639b9bc-4d27-472f-91d9-be13bab2be96","Type":"ContainerStarted","Data":"7a2015d46b4ac83500984abb62e7e26a4338988f1c15b2003e25b29b85d86c2c"}
Oct 31 00:04:55.754677 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:55.754562    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v" event={"ID":"6639b9bc-4d27-472f-91d9-be13bab2be96","Type":"ContainerStarted","Data":"d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24"}
Oct 31 00:04:55.755055 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:55.754690    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:04:55.869542 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:04:55.869478    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v" podStartSLOduration=1.9827363180000002 podStartE2EDuration="2.869458429s" totalImagesPullingTime="886.722111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:55.867716504 +0000 UTC m=+1401.158264823" watchObservedRunningTime="2026-10-31 00:04:55.869458429 +0000 UTC m=+1401.160006753"
Oct 31 00:05:01.766335 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:05:01.766296    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:05:08.319516 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:05:08.319471    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:05:38.374791 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:05:38.374745    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:06:08.325858 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:06:08.325815    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:06:38.322917 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:06:38.322873    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:06:43.563741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:43.563701    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"]
Oct 31 00:06:43.564228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:43.563946    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v" podUID="6639b9bc-4d27-472f-91d9-be13bab2be96" containerName="maas-api" containerID="cri-o://d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24" gracePeriod=30
Oct 31 00:06:43.878679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:43.878618    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:06:44.022306 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.022117    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls\") pod \"6639b9bc-4d27-472f-91d9-be13bab2be96\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") "
Oct 31 00:06:44.022599 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.022575    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca\") pod \"6639b9bc-4d27-472f-91d9-be13bab2be96\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") "
Oct 31 00:06:44.022772 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.022757    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs\") pod \"6639b9bc-4d27-472f-91d9-be13bab2be96\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") "
Oct 31 00:06:44.023085 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.023061    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh\" (UniqueName: \"kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh\") pod \"6639b9bc-4d27-472f-91d9-be13bab2be96\" (UID: \"6639b9bc-4d27-472f-91d9-be13bab2be96\") "
Oct 31 00:06:44.027800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.027412    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh" pod "6639b9bc-4d27-472f-91d9-be13bab2be96" (UID: "6639b9bc-4d27-472f-91d9-be13bab2be96"). InnerVolumeSpecName "kube-api-access-ld8rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:44.047682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.041282    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs" pod "6639b9bc-4d27-472f-91d9-be13bab2be96" (UID: "6639b9bc-4d27-472f-91d9-be13bab2be96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:44.047682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.044005    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca" pod "6639b9bc-4d27-472f-91d9-be13bab2be96" (UID: "6639b9bc-4d27-472f-91d9-be13bab2be96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:44.048864 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.048821    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls" pod "6639b9bc-4d27-472f-91d9-be13bab2be96" (UID: "6639b9bc-4d27-472f-91d9-be13bab2be96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:44.127354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.127267    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8rh\" (UniqueName: \"kubernetes.io/projected/6639b9bc-4d27-472f-91d9-be13bab2be96-kube-api-access-ld8rh\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:44.127354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.127312    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:44.127354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.127331    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6639b9bc-4d27-472f-91d9-be13bab2be96-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:44.127354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.127348    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6639b9bc-4d27-472f-91d9-be13bab2be96-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:44.521295 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.521201    4466 generic.go:356] "Generic (PLEG): container finished" podID="6639b9bc-4d27-472f-91d9-be13bab2be96" containerID="d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24" exitCode=0
Oct 31 00:06:44.521295 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.521268    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"
Oct 31 00:06:44.521295 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.521279    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v" event={"ID":"6639b9bc-4d27-472f-91d9-be13bab2be96","Type":"ContainerDied","Data":"d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24"}
Oct 31 00:06:44.521527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.521368    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v" event={"ID":"6639b9bc-4d27-472f-91d9-be13bab2be96","Type":"ContainerDied","Data":"7a2015d46b4ac83500984abb62e7e26a4338988f1c15b2003e25b29b85d86c2c"}
Oct 31 00:06:44.521527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.521400    4466 scope.go:122] "RemoveContainer" containerID="d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24"
Oct 31 00:06:44.534299 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.534275    4466 scope.go:122] "RemoveContainer" containerID="d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24"
Oct 31 00:06:44.534615 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:06:44.534592    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24\": container with ID starting with d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24 not found: ID does not exist" containerID="d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24"
Oct 31 00:06:44.534719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.534625    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24"} err="failed to get container status \"d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24\": rpc error: code = NotFound desc = could not find container \"d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24\": container with ID starting with d2437f0c4fcc21cc29609ccd4232b880f0c1276ff3d9ff8c30b6356dbbd7fa24 not found: ID does not exist"
Oct 31 00:06:44.682780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.682747    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"]
Oct 31 00:06:44.727576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.727546    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d32a7cc0-6465645fbc-cxt6v"]
Oct 31 00:06:44.828904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:44.828834    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6639b9bc-4d27-472f-91d9-be13bab2be96" path="/var/lib/kubelet/pods/6639b9bc-4d27-472f-91d9-be13bab2be96/volumes"
Oct 31 00:06:47.818905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:47.818865    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"]
Oct 31 00:06:47.819397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:47.819108    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy" containerID="cri-o://fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365" gracePeriod=30
Oct 31 00:06:47.865851 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:47.865810    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"]
Oct 31 00:06:47.866133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:47.866107    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy" containerID="cri-o://5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0" gracePeriod=30
Oct 31 00:06:53.157944 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.157914    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 31 00:06:53.193811 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.193781    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:06:53.274071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274039    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274121    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.274251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274171    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274222    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274280    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.274450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274331    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274336    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.274450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274360    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.274450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274388    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.274450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274408    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274460    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274495    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274525    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274515    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274560    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274706    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.274777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274747    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274801    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274861    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274560    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274578    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274823    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274844    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274892    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.274966    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token\") pod \"086adb9a-4bca-44a8-ad31-051f0219b802\" (UID: \"086adb9a-4bca-44a8-ad31-051f0219b802\") "
Oct 31 00:06:53.275109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.275039    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy\") pod \"6518a1b2-9688-417d-a45d-3ac2e9b98967\" (UID: \"6518a1b2-9688-417d-a45d-3ac2e9b98967\") "
Oct 31 00:06:53.275567 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.275244    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:53.275567 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.275492    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.275975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.275948    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.275982    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.275998    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276012    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276027    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276041    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6518a1b2-9688-417d-a45d-3ac2e9b98967-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276055    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086adb9a-4bca-44a8-ad31-051f0219b802-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276070    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276086    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.276514 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.276101    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.277246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.277219    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:53.277676 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.277618    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:06:53.277767 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.277703    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "kube-api-access-4tzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:53.277767 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.277744    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:06:53.277949 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.277930    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.277949 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.277937    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:53.278216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.278195    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm" pod "086adb9a-4bca-44a8-ad31-051f0219b802" (UID: "086adb9a-4bca-44a8-ad31-051f0219b802"). InnerVolumeSpecName "kube-api-access-8v2zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:53.278472 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.278459    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy" pod "6518a1b2-9688-417d-a45d-3ac2e9b98967" (UID: "6518a1b2-9688-417d-a45d-3ac2e9b98967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:53.377606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377510    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377542    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086adb9a-4bca-44a8-ad31-051f0219b802-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377557    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086adb9a-4bca-44a8-ad31-051f0219b802-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377571    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2zm\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-kube-api-access-8v2zm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377585    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377598    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tzt2\" (UniqueName: \"kubernetes.io/projected/6518a1b2-9688-417d-a45d-3ac2e9b98967-kube-api-access-4tzt2\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377992 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377611    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086adb9a-4bca-44a8-ad31-051f0219b802-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.377992 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.377623    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6518a1b2-9688-417d-a45d-3ac2e9b98967-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:53.582845 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.582798    4466 generic.go:356] "Generic (PLEG): container finished" podID="086adb9a-4bca-44a8-ad31-051f0219b802" containerID="fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365" exitCode=0
Oct 31 00:06:53.583020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.582911    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"
Oct 31 00:06:53.583020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.582932    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" event={"ID":"086adb9a-4bca-44a8-ad31-051f0219b802","Type":"ContainerDied","Data":"fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365"}
Oct 31 00:06:53.583020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.582988    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch" event={"ID":"086adb9a-4bca-44a8-ad31-051f0219b802","Type":"ContainerDied","Data":"a2f624dc235b1ba2cd0875cfa7d573fffc5a66b461e4345223d77b1188f8ea10"}
Oct 31 00:06:53.583020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.583010    4466 scope.go:122] "RemoveContainer" containerID="fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365"
Oct 31 00:06:53.584615 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.584585    4466 generic.go:356] "Generic (PLEG): container finished" podID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerID="5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0" exitCode=0
Oct 31 00:06:53.584789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.584663    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" event={"ID":"6518a1b2-9688-417d-a45d-3ac2e9b98967","Type":"ContainerDied","Data":"5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0"}
Oct 31 00:06:53.584789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.584684    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"
Oct 31 00:06:53.584789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.584739    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695" event={"ID":"6518a1b2-9688-417d-a45d-3ac2e9b98967","Type":"ContainerDied","Data":"47da4a63cfd8bfc72250abbd1a871ecf8452ea4de1c6f8c3931fc79008a882cf"}
Oct 31 00:06:53.594584 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.594562    4466 scope.go:122] "RemoveContainer" containerID="fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365"
Oct 31 00:06:53.594951 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:06:53.594933    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365\": container with ID starting with fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365 not found: ID does not exist" containerID="fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365"
Oct 31 00:06:53.595043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.594959    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365"} err="failed to get container status \"fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365\": rpc error: code = NotFound desc = could not find container \"fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365\": container with ID starting with fb26b7f8d2e846d5633ffc1e79d27fc66d2c8d602b23d2d95eb540652645b365 not found: ID does not exist"
Oct 31 00:06:53.595043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.594976    4466 scope.go:122] "RemoveContainer" containerID="5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0"
Oct 31 00:06:53.605594 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.605570    4466 scope.go:122] "RemoveContainer" containerID="5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0"
Oct 31 00:06:53.605940 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:06:53.605918    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0\": container with ID starting with 5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0 not found: ID does not exist" containerID="5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0"
Oct 31 00:06:53.605996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.605951    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0"} err="failed to get container status \"5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0\": rpc error: code = NotFound desc = could not find container \"5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0\": container with ID starting with 5098d9988d67ff4ccbb575c1e4ce0143df03af151efba425daef5be89d757cc0 not found: ID does not exist"
Oct 31 00:06:53.748006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.747915    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"]
Oct 31 00:06:53.752041 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.752013    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-84bch"]
Oct 31 00:06:53.867990 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.867959    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"]
Oct 31 00:06:53.872701 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:53.872671    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-bn695"]
Oct 31 00:06:54.889080 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:54.886803    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" path="/var/lib/kubelet/pods/086adb9a-4bca-44a8-ad31-051f0219b802/volumes"
Oct 31 00:06:54.892154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:54.889772    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" path="/var/lib/kubelet/pods/6518a1b2-9688-417d-a45d-3ac2e9b98967/volumes"
Oct 31 00:06:59.942811 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.942779    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"]
Oct 31 00:06:59.943369 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943356    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy"
Oct 31 00:06:59.943430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943370    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy"
Oct 31 00:06:59.943430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943388    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6639b9bc-4d27-472f-91d9-be13bab2be96" containerName="maas-api"
Oct 31 00:06:59.943430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943393    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6639b9bc-4d27-472f-91d9-be13bab2be96" containerName="maas-api"
Oct 31 00:06:59.943430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943419    4466 cpu_manager.go:414] "containerMap: removing container" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy"
Oct 31 00:06:59.943430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943424    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy"
Oct 31 00:06:59.943583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943513    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="086adb9a-4bca-44a8-ad31-051f0219b802" containerName="istio-proxy"
Oct 31 00:06:59.943583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943525    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6518a1b2-9688-417d-a45d-3ac2e9b98967" containerName="istio-proxy"
Oct 31 00:06:59.943583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.943532    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6639b9bc-4d27-472f-91d9-be13bab2be96" containerName="maas-api"
Oct 31 00:06:59.947105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.947086    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:06:59.996493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:06:59.996458    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a251fdfb-openshift-default-dockercfg-k8cql\""
Oct 31 00:07:00.033424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033375    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.033424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033431    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.033752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033538    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.033752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033602    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.033752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033718    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.033927 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033800    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnlh\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.033927 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033879    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.034032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.033937    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.034032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.034016    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.049354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.049315    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"]
Oct 31 00:07:00.135177 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135144    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135384 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135240    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135384 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135263    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135384 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135306    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135570 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135393    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135570 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135435    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135570 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135494    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnlh\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135570 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135560    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135658    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.135870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.135806    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.136829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.136033    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.136829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.136049    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.136829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.136387    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.136829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.136423    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.138801 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.138769    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.139386 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.139367    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.197067 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.196976    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.199389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.199361    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnlh\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.272021 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.271975    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:00.816383 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.816351    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"]
Oct 31 00:07:00.817821 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:07:00.817794    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241e425b_d2c2_4819_b417_a17a1266b094.slice/crio-0cc4290703db305be68cbee9e1fea3f82007bf22160e076e8f43e0684ed213a0 WatchSource:0}: Error finding container 0cc4290703db305be68cbee9e1fea3f82007bf22160e076e8f43e0684ed213a0: Status 404 returned error can't find the container with id 0cc4290703db305be68cbee9e1fea3f82007bf22160e076e8f43e0684ed213a0
Oct 31 00:07:00.821001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.820960    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:07:00.821147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.821053    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:07:00.821147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:00.821098    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:07:01.644182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:01.644147    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" event={"ID":"241e425b-d2c2-4819-b417-a17a1266b094","Type":"ContainerStarted","Data":"492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"}
Oct 31 00:07:01.644598 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:01.644202    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" event={"ID":"241e425b-d2c2-4819-b417-a17a1266b094","Type":"ContainerStarted","Data":"0cc4290703db305be68cbee9e1fea3f82007bf22160e076e8f43e0684ed213a0"}
Oct 31 00:07:01.759302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:01.759244    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podStartSLOduration=2.759228654 podStartE2EDuration="2.759228654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:01.757179987 +0000 UTC m=+1527.047728307" watchObservedRunningTime="2026-10-31 00:07:01.759228654 +0000 UTC m=+1527.049776973"
Oct 31 00:07:02.272565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:02.272530    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:02.283013 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:02.282982    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:02.651351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:02.651318    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:02.652439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:02.652417    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:07:05.063012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.062976    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"]
Oct 31 00:07:05.067379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.067359    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.115854 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.115807    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.116079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.116050    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s678g\" (UniqueName: \"kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.116195 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.116174    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.116236 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.116209    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.124199 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.124166    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"]
Oct 31 00:07:05.166410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.166370    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7c8802f8-serving-cert\""
Oct 31 00:07:05.217905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.217875    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.217905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.217915    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.218175 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.218019    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.218175 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:07:05.218143    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-serving-cert: secret "maas-api-e2e-ait-7c8802f8-serving-cert" not found
Oct 31 00:07:05.218293 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:07:05.218233    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls podName:7b4d7ba2-4767-40c7-9009-9e4b9b91101d nodeName:}" failed. No retries permitted until 2026-10-31 00:07:05.718207916 +0000 UTC m=+1531.008756222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls") pod "maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" (UID: "7b4d7ba2-4767-40c7-9009-9e4b9b91101d") : secret "maas-api-e2e-ait-7c8802f8-serving-cert" not found
Oct 31 00:07:05.218362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.218290    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s678g\" (UniqueName: \"kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.218811 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.218774    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.221166 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.221140    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.278689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.278647    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s678g\" (UniqueName: \"kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.736939 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.736904    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.739870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.739836    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\") pod \"maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:05.983195 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:05.983155    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:06.366092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:06.366059    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"]
Oct 31 00:07:06.367342 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:07:06.367314    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4d7ba2_4767_40c7_9009_9e4b9b91101d.slice/crio-c582d8d6ac3c684c68efc16efcdbc8b19a8f8c282338cf31304075bdc966fa4d WatchSource:0}: Error finding container c582d8d6ac3c684c68efc16efcdbc8b19a8f8c282338cf31304075bdc966fa4d: Status 404 returned error can't find the container with id c582d8d6ac3c684c68efc16efcdbc8b19a8f8c282338cf31304075bdc966fa4d
Oct 31 00:07:06.678673 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:06.678561    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" event={"ID":"7b4d7ba2-4767-40c7-9009-9e4b9b91101d","Type":"ContainerStarted","Data":"c582d8d6ac3c684c68efc16efcdbc8b19a8f8c282338cf31304075bdc966fa4d"}
Oct 31 00:07:07.686330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:07.686293    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" event={"ID":"7b4d7ba2-4767-40c7-9009-9e4b9b91101d","Type":"ContainerStarted","Data":"9be519fd9be6b14bbc4a10fefaa16b45c9d4b32da79122b3c1aa0b4d475e44fa"}
Oct 31 00:07:07.686804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:07.686390    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:07.799672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:07.799596    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" podStartSLOduration=1.904909322 podStartE2EDuration="2.79958056s" totalImagesPullingTime="894.671238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:07.797704522 +0000 UTC m=+1533.088252845" watchObservedRunningTime="2026-10-31 00:07:07.79958056 +0000 UTC m=+1533.090128879"
Oct 31 00:07:08.372382 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:07:08.372338    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:07:13.700677 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:07:13.699428    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:07:38.318406 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:07:38.318361    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:08:08.318763 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:08:08.318721    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:08:21.946768 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:21.946731    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"]
Oct 31 00:08:21.949255 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:21.946992    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj" podUID="d7683bc7-a477-45aa-ac74-e0759935f599" containerName="maas-api" containerID="cri-o://c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147" gracePeriod=30
Oct 31 00:08:22.261559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.261529    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:08:22.294368 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.294337    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca\") pod \"d7683bc7-a477-45aa-ac74-e0759935f599\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") "
Oct 31 00:08:22.294566 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.294437    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\") pod \"d7683bc7-a477-45aa-ac74-e0759935f599\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") "
Oct 31 00:08:22.294646 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.294571    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs\") pod \"d7683bc7-a477-45aa-ac74-e0759935f599\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") "
Oct 31 00:08:22.294707 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.294665    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc\" (UniqueName: \"kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc\") pod \"d7683bc7-a477-45aa-ac74-e0759935f599\" (UID: \"d7683bc7-a477-45aa-ac74-e0759935f599\") "
Oct 31 00:08:22.295281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.294940    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca" pod "d7683bc7-a477-45aa-ac74-e0759935f599" (UID: "d7683bc7-a477-45aa-ac74-e0759935f599"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:08:22.296470 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.295869    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7683bc7-a477-45aa-ac74-e0759935f599-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:22.297229 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.297188    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls" pod "d7683bc7-a477-45aa-ac74-e0759935f599" (UID: "d7683bc7-a477-45aa-ac74-e0759935f599"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:08:22.297346 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.297259    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc" pod "d7683bc7-a477-45aa-ac74-e0759935f599" (UID: "d7683bc7-a477-45aa-ac74-e0759935f599"). InnerVolumeSpecName "kube-api-access-ftsdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:22.298767 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.298744    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs" pod "d7683bc7-a477-45aa-ac74-e0759935f599" (UID: "d7683bc7-a477-45aa-ac74-e0759935f599"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:08:22.389924 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.389876    4466 generic.go:356] "Generic (PLEG): container finished" podID="d7683bc7-a477-45aa-ac74-e0759935f599" containerID="c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147" exitCode=0
Oct 31 00:08:22.390122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.389939    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"
Oct 31 00:08:22.390122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.389967    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj" event={"ID":"d7683bc7-a477-45aa-ac74-e0759935f599","Type":"ContainerDied","Data":"c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147"}
Oct 31 00:08:22.390122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.390030    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj" event={"ID":"d7683bc7-a477-45aa-ac74-e0759935f599","Type":"ContainerDied","Data":"7dbf352fbe7ce7e335594d6cfc23400cb2079e421e74866f6a08d64400e27a10"}
Oct 31 00:08:22.390122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.390062    4466 scope.go:122] "RemoveContainer" containerID="c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147"
Oct 31 00:08:22.397528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.397503    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:22.397528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.397526    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7683bc7-a477-45aa-ac74-e0759935f599-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:22.397528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.397542    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsdc\" (UniqueName: \"kubernetes.io/projected/d7683bc7-a477-45aa-ac74-e0759935f599-kube-api-access-ftsdc\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:22.420679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.420564    4466 scope.go:122] "RemoveContainer" containerID="c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147"
Oct 31 00:08:22.421892 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:08:22.421666    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147\": container with ID starting with c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147 not found: ID does not exist" containerID="c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147"
Oct 31 00:08:22.422373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.422345    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147"} err="failed to get container status \"c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147\": rpc error: code = NotFound desc = could not find container \"c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147\": container with ID starting with c3a58a01eb3e1827b574462688f52bcc81f6809cb128f28ffcdc85fa6e826147 not found: ID does not exist"
Oct 31 00:08:22.567898 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.566676    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"]
Oct 31 00:08:22.570763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.570730    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0751b65-7447786c8f-425lj"]
Oct 31 00:08:22.828574 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:22.828493    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7683bc7-a477-45aa-ac74-e0759935f599" path="/var/lib/kubelet/pods/d7683bc7-a477-45aa-ac74-e0759935f599/volumes"
Oct 31 00:08:38.320181 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:08:38.320143    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:08:48.639761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:48.639702    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"]
Oct 31 00:08:48.641187 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:48.641142    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" containerID="cri-o://1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64" gracePeriod=30
Oct 31 00:08:51.393026 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:51.392982    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:08:51.408739 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:51.408699    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 00:08:51.723183 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:51.723076    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 00:08:51.897791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:51.897732    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:08:51.916802 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:51.916754    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:08:51.976835 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:51.976728    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:08:52.910333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:52.910296    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:08:52.910839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:52.910360    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:08:54.353691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.353123    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490242    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490298    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490341    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490408    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490492    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490568    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490604    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490685    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.490791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.490755    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo\") pod \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\" (UID: \"bfd37248-4e95-4847-a9b7-07d94e99aa4a\") "
Oct 31 00:08:54.492026 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.491727    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:08:54.492452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.492431    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:54.493035 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.493003    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:54.493302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.493286    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:54.493561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.493543    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:54.494102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.494071    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:54.496503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.496478    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "kube-api-access-nwrn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:54.496602 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.496577    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:54.503934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.503892    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo" pod "bfd37248-4e95-4847-a9b7-07d94e99aa4a" (UID: "bfd37248-4e95-4847-a9b7-07d94e99aa4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:08:54.593147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593102    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593138    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrn6\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-kube-api-access-nwrn6\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593149    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593158    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593167    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593177    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593185    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593202    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.593519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.593211    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd37248-4e95-4847-a9b7-07d94e99aa4a-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:54.764715 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.764381    4466 generic.go:356] "Generic (PLEG): container finished" podID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerID="1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64" exitCode=0
Oct 31 00:08:54.774722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.765031    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" event={"ID":"bfd37248-4e95-4847-a9b7-07d94e99aa4a","Type":"ContainerDied","Data":"1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64"}
Oct 31 00:08:54.774722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.765111    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8" event={"ID":"bfd37248-4e95-4847-a9b7-07d94e99aa4a","Type":"ContainerDied","Data":"d6366bdde5feb9ed0672b994d25298c82c82babc53adaefe43465b234f4f5003"}
Oct 31 00:08:54.774722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.765144    4466 scope.go:122] "RemoveContainer" containerID="1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64"
Oct 31 00:08:54.774722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.765318    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"
Oct 31 00:08:54.918143 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.907228    4466 scope.go:122] "RemoveContainer" containerID="1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64"
Oct 31 00:08:54.918143 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:08:54.908309    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64\": container with ID starting with 1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64 not found: ID does not exist" containerID="1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64"
Oct 31 00:08:54.918143 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.908350    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64"} err="failed to get container status \"1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64\": rpc error: code = NotFound desc = could not find container \"1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64\": container with ID starting with 1a7eb53a5cc4df2e43134b578433a969dc47e61831d1d39956afeb9500b05b64 not found: ID does not exist"
Oct 31 00:08:54.967845 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.967808    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"]
Oct 31 00:08:54.978117 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:54.976280    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-4jhn8"]
Oct 31 00:08:56.830395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:56.830360    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" path="/var/lib/kubelet/pods/bfd37248-4e95-4847-a9b7-07d94e99aa4a/volumes"
Oct 31 00:08:57.785853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:57.785816    4466 generic.go:356] "Generic (PLEG): container finished" podID="241e425b-d2c2-4819-b417-a17a1266b094" containerID="492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868" exitCode=0
Oct 31 00:08:57.786079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:57.785886    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" event={"ID":"241e425b-d2c2-4819-b417-a17a1266b094","Type":"ContainerDied","Data":"492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"}
Oct 31 00:08:57.786360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:57.786346    4466 scope.go:122] "RemoveContainer" containerID="492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"
Oct 31 00:08:57.788043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:57.788018    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:08:57.788130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:57.788086    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:08:57.788130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:57.788123    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:08:58.796341 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:58.796307    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" event={"ID":"241e425b-d2c2-4819-b417-a17a1266b094","Type":"ContainerStarted","Data":"f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181"}
Oct 31 00:08:59.273143 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:59.273105    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:08:59.274541 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:59.274512    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:08:59.274677 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:08:59.274573    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:09:00.272830 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:00.272795    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:09:00.273264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:00.272851    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:09:01.272756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:01.272724    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:09:01.273017 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:01.272990    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:09:02.272690 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:02.272626    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:09:02.272841 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:02.272724    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:09:02.652816 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:02.652755    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:09:03.272715 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.272678    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:09:03.272896 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.272731    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:09:03.540761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.539455    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"]
Oct 31 00:09:03.540761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.539741    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" podUID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" containerName="maas-api" containerID="cri-o://9be519fd9be6b14bbc4a10fefaa16b45c9d4b32da79122b3c1aa0b4d475e44fa" gracePeriod=30
Oct 31 00:09:03.724176 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.724134    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" podUID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.78:8443/health\": dial tcp 10.133.0.78:8443: connect: connection refused"
Oct 31 00:09:03.894232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.894188    4466 generic.go:356] "Generic (PLEG): container finished" podID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" containerID="9be519fd9be6b14bbc4a10fefaa16b45c9d4b32da79122b3c1aa0b4d475e44fa" exitCode=0
Oct 31 00:09:03.894436 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:03.894309    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" event={"ID":"7b4d7ba2-4767-40c7-9009-9e4b9b91101d","Type":"ContainerDied","Data":"9be519fd9be6b14bbc4a10fefaa16b45c9d4b32da79122b3c1aa0b4d475e44fa"}
Oct 31 00:09:04.103977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.103953    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:09:04.298714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.297579    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g\" (UniqueName: \"kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g\") pod \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") "
Oct 31 00:09:04.298714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.297680    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\") pod \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") "
Oct 31 00:09:04.298714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.297848    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca\") pod \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") "
Oct 31 00:09:04.298714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.297897    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs\") pod \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\" (UID: \"7b4d7ba2-4767-40c7-9009-9e4b9b91101d\") "
Oct 31 00:09:04.309543 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.308113    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:09:04.309543 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.308172    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 00:09:04.320687 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.317546    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs" pod "7b4d7ba2-4767-40c7-9009-9e4b9b91101d" (UID: "7b4d7ba2-4767-40c7-9009-9e4b9b91101d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:09:04.320687 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.318363    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca" pod "7b4d7ba2-4767-40c7-9009-9e4b9b91101d" (UID: "7b4d7ba2-4767-40c7-9009-9e4b9b91101d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:09:04.327364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.327291    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls" pod "7b4d7ba2-4767-40c7-9009-9e4b9b91101d" (UID: "7b4d7ba2-4767-40c7-9009-9e4b9b91101d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:09:04.333756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.333718    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g" pod "7b4d7ba2-4767-40c7-9009-9e4b9b91101d" (UID: "7b4d7ba2-4767-40c7-9009-9e4b9b91101d"). InnerVolumeSpecName "kube-api-access-s678g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:04.399932 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.399875    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s678g\" (UniqueName: \"kubernetes.io/projected/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-kube-api-access-s678g\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:04.400181 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.400164    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:04.400285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.400272    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:04.400381 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.400371    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b4d7ba2-4767-40c7-9009-9e4b9b91101d-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:04.907282 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.907250    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"
Oct 31 00:09:04.907282 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.907258    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8" event={"ID":"7b4d7ba2-4767-40c7-9009-9e4b9b91101d","Type":"ContainerDied","Data":"c582d8d6ac3c684c68efc16efcdbc8b19a8f8c282338cf31304075bdc966fa4d"}
Oct 31 00:09:04.907891 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:04.907333    4466 scope.go:122] "RemoveContainer" containerID="9be519fd9be6b14bbc4a10fefaa16b45c9d4b32da79122b3c1aa0b4d475e44fa"
Oct 31 00:09:05.077667 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:05.077612    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"]
Oct 31 00:09:05.095093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:05.095009    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c8802f8-7bfc6c7cf-kgfw8"]
Oct 31 00:09:05.279225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:05.279183    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:09:05.282784 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:05.280993    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:09:06.828806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:06.828773    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" path="/var/lib/kubelet/pods/7b4d7ba2-4767-40c7-9009-9e4b9b91101d/volumes"
Oct 31 00:09:08.366368 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:09:08.366320    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:09:32.667736 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:32.667702    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"]
Oct 31 00:09:32.668224 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:32.667935    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" containerID="cri-o://8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99" gracePeriod=30
Oct 31 00:09:37.457656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:37.453703    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:09:37.457656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:37.453783    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:09:38.066054 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.066018    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:09:38.108959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.108928    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109155 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109036    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109155 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109124    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109171    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109193    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109229    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109260    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.109439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109289    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109332    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.109591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109466    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109502    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy\") pod \"4e5ed964-d641-484e-90c3-2830a086604c\" (UID: \"4e5ed964-d641-484e-90c3-2830a086604c\") "
Oct 31 00:09:38.109763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109678    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.109869 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.109846    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.110380 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.110354    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.110491 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.110389    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.110491 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.110406    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.110491 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.110420    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.110702 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.110606    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:09:38.111768 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.111743    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:38.111893 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.111791    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:09:38.111893 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.111876    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "kube-api-access-cpg4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:38.112063 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.111995    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy" pod "4e5ed964-d641-484e-90c3-2830a086604c" (UID: "4e5ed964-d641-484e-90c3-2830a086604c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.201365 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.201330    4466 generic.go:356] "Generic (PLEG): container finished" podID="4e5ed964-d641-484e-90c3-2830a086604c" containerID="8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99" exitCode=0
Oct 31 00:09:38.201536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.201385    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"
Oct 31 00:09:38.201536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.201410    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" event={"ID":"4e5ed964-d641-484e-90c3-2830a086604c","Type":"ContainerDied","Data":"8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99"}
Oct 31 00:09:38.201536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.201487    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2" event={"ID":"4e5ed964-d641-484e-90c3-2830a086604c","Type":"ContainerDied","Data":"0c1c1223991abd932b674525d00b765e07ec59c9d4be2deeb42687d37c35f578"}
Oct 31 00:09:38.201536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.201509    4466 scope.go:122] "RemoveContainer" containerID="8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99"
Oct 31 00:09:38.212124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.212005    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5ed964-d641-484e-90c3-2830a086604c-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.212124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.212034    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg4w\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-kube-api-access-cpg4w\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.212124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.212054    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5ed964-d641-484e-90c3-2830a086604c-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.212124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.212069    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5ed964-d641-484e-90c3-2830a086604c-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.212124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.212085    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5ed964-d641-484e-90c3-2830a086604c-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.214675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.213373    4466 scope.go:122] "RemoveContainer" containerID="8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99"
Oct 31 00:09:38.214675 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:09:38.213733    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99\": container with ID starting with 8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99 not found: ID does not exist" containerID="8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99"
Oct 31 00:09:38.214675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.213767    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99"} err="failed to get container status \"8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99\": rpc error: code = NotFound desc = could not find container \"8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99\": container with ID starting with 8e4943f02ae7dcc6639b03fd6c786b6c45af974fd23b3f66ef14a7e220a5bd99 not found: ID does not exist"
Oct 31 00:09:38.321390 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:09:38.321294    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:09:38.338446 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.338399    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"]
Oct 31 00:09:38.344826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.344701    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-px9v2"]
Oct 31 00:09:38.828037 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:09:38.828007    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5ed964-d641-484e-90c3-2830a086604c" path="/var/lib/kubelet/pods/4e5ed964-d641-484e-90c3-2830a086604c/volumes"
Oct 31 00:10:08.317988 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:10:08.317955    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:10:38.317225 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:10:38.317185    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:11:08.321025 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:11:08.320984    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:11:33.728642 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:33.728596    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"]
Oct 31 00:11:33.729172 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:33.728848    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" containerID="cri-o://603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67" gracePeriod=30
Oct 31 00:11:33.775164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:33.775129    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"]
Oct 31 00:11:33.775371 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:33.775328    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" containerID="cri-o://4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2" gracePeriod=30
Oct 31 00:11:36.533822 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:36.533785    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:11:36.534240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:36.533844    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:11:38.318616 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:11:38.318573    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:11:38.394828 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:38.394791    4466 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:11:38.394992 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:38.394858    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:11:39.053656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.053338    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:11:39.098820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.098795    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 31 00:11:39.110957 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.110927    4466 generic.go:356] "Generic (PLEG): container finished" podID="200386c3-07db-4689-ba2d-ea12c713b4df" containerID="4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2" exitCode=0
Oct 31 00:11:39.111133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.110975    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"
Oct 31 00:11:39.111133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.111012    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" event={"ID":"200386c3-07db-4689-ba2d-ea12c713b4df","Type":"ContainerDied","Data":"4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2"}
Oct 31 00:11:39.111133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.111078    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8" event={"ID":"200386c3-07db-4689-ba2d-ea12c713b4df","Type":"ContainerDied","Data":"830276ad5d7242036147e630c2bd58ebaa93186d73804deb1292ec15902402d8"}
Oct 31 00:11:39.111133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.111108    4466 scope.go:122] "RemoveContainer" containerID="4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2"
Oct 31 00:11:39.112507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.112487    4466 generic.go:356] "Generic (PLEG): container finished" podID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerID="603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67" exitCode=0
Oct 31 00:11:39.112612 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.112534    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"
Oct 31 00:11:39.112612 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.112537    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" event={"ID":"e42004e9-c4d2-435a-8f4b-0c283473b484","Type":"ContainerDied","Data":"603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67"}
Oct 31 00:11:39.112612 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.112583    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb" event={"ID":"e42004e9-c4d2-435a-8f4b-0c283473b484","Type":"ContainerDied","Data":"f7435768a5d7ca8f738e62e7fd751d8d6c3391f76ae1f9f3232009bd6294957b"}
Oct 31 00:11:39.120855 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.120831    4466 scope.go:122] "RemoveContainer" containerID="4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2"
Oct 31 00:11:39.121129 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:11:39.121108    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2\": container with ID starting with 4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2 not found: ID does not exist" containerID="4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2"
Oct 31 00:11:39.121188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.121143    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2"} err="failed to get container status \"4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2\": rpc error: code = NotFound desc = could not find container \"4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2\": container with ID starting with 4afb11ccf11265678995af07490402d769474135fa94a06a98de921cc11906e2 not found: ID does not exist"
Oct 31 00:11:39.121188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.121160    4466 scope.go:122] "RemoveContainer" containerID="603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67"
Oct 31 00:11:39.130178 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.130161    4466 scope.go:122] "RemoveContainer" containerID="603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67"
Oct 31 00:11:39.130447 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:11:39.130425    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67\": container with ID starting with 603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67 not found: ID does not exist" containerID="603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67"
Oct 31 00:11:39.130521 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.130458    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67"} err="failed to get container status \"603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67\": rpc error: code = NotFound desc = could not find container \"603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67\": container with ID starting with 603bc99f6b9e9be2dc8a4f13b5c97af633ff73efc2f4f88e3da23652c0f66c67 not found: ID does not exist"
Oct 31 00:11:39.173063 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173028    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.173269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173080    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.173269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173152    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.173269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173194    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.173269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173222    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173296    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173348    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173385    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173452    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173512    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173611    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173665    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173726    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173756    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173790    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173822    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173842    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173913    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token\") pod \"e42004e9-c4d2-435a-8f4b-0c283473b484\" (UID: \"e42004e9-c4d2-435a-8f4b-0c283473b484\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173963    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.173995    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo\") pod \"200386c3-07db-4689-ba2d-ea12c713b4df\" (UID: \"200386c3-07db-4689-ba2d-ea12c713b4df\") "
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.174049    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.174468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.174166    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:39.175268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.174872    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.175268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.174895    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.175268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.174910    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42004e9-c4d2-435a-8f4b-0c283473b484-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.175268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.174926    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.175712 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.175416    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.178225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.177986    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.180591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180171    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.180591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180430    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.180591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180445    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.180591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180513    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:11:39.180591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180569    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.180979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180670    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl" pod "e42004e9-c4d2-435a-8f4b-0c283473b484" (UID: "e42004e9-c4d2-435a-8f4b-0c283473b484"). InnerVolumeSpecName "kube-api-access-p5nwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.180979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.180954    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:39.181135 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.181110    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.185332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.185295    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:11:39.185438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.185336    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "kube-api-access-zvpk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.185863 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.185842    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.186351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.186311    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token" pod "200386c3-07db-4689-ba2d-ea12c713b4df" (UID: "200386c3-07db-4689-ba2d-ea12c713b4df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.276597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276502    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.276597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276541    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.276597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276557    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.276597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276571    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5nwl\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-kube-api-access-p5nwl\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.276597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276584    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.276597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276597    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276610    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276622    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276660    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276676    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200386c3-07db-4689-ba2d-ea12c713b4df-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276688    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42004e9-c4d2-435a-8f4b-0c283473b484-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276701    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpk9\" (UniqueName: \"kubernetes.io/projected/200386c3-07db-4689-ba2d-ea12c713b4df-kube-api-access-zvpk9\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276717    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200386c3-07db-4689-ba2d-ea12c713b4df-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.277014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.276730    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200386c3-07db-4689-ba2d-ea12c713b4df-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.529006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.528921    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"]
Oct 31 00:11:39.533187 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.533155    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-qsdr8"]
Oct 31 00:11:39.640483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.640449    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"]
Oct 31 00:11:39.646234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:39.646187    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-dxpzb"]
Oct 31 00:11:40.216963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.216903    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"]
Oct 31 00:11:40.217523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217503    4466 cpu_manager.go:414] "containerMap: removing container" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy"
Oct 31 00:11:40.217523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217522    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217550    4466 cpu_manager.go:414] "containerMap: removing container" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217555    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217571    4466 cpu_manager.go:414] "containerMap: removing container" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217576    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217584    4466 cpu_manager.go:414] "containerMap: removing container" podUID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" containerName="maas-api"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217588    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" containerName="maas-api"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217596    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d7683bc7-a477-45aa-ac74-e0759935f599" containerName="maas-api"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217601    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7683bc7-a477-45aa-ac74-e0759935f599" containerName="maas-api"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217616    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217621    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217733    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217744    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5ed964-d641-484e-90c3-2830a086604c" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217755    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd37248-4e95-4847-a9b7-07d94e99aa4a" containerName="istio-proxy"
Oct 31 00:11:40.217756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217761    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" containerName="istio-proxy"
Oct 31 00:11:40.218221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217768    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4d7ba2-4767-40c7-9009-9e4b9b91101d" containerName="maas-api"
Oct 31 00:11:40.218221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.217773    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7683bc7-a477-45aa-ac74-e0759935f599" containerName="maas-api"
Oct 31 00:11:40.222604 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.222580    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.272198 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.272166    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-4a3dba\""
Oct 31 00:11:40.272565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.272535    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-4a3dba-dockercfg-k7npz\""
Oct 31 00:11:40.285660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.276429    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"]
Oct 31 00:11:40.312512 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.312468    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz85w\" (UniqueName: \"kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w\") pod \"payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.312853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.312822    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.375880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.372346    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"]
Oct 31 00:11:40.377840 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.377805    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.415701 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.415534    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz85w\" (UniqueName: \"kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w\") pod \"payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.415701 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.415596    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.419652 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.416823    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.445700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.441916    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"]
Oct 31 00:11:40.517713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.517125    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5qs\" (UniqueName: \"kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs\") pod \"payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.517713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.517279    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.541659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.519661    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz85w\" (UniqueName: \"kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w\") pod \"payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.546683 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.545599    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:40.620321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.620280    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.620575 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.620550    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5qs\" (UniqueName: \"kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs\") pod \"payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.621901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.621863    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.686889 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.686860    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5qs\" (UniqueName: \"kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs\") pod \"payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.694581 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.694550    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:40.836764 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.836735    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200386c3-07db-4689-ba2d-ea12c713b4df" path="/var/lib/kubelet/pods/200386c3-07db-4689-ba2d-ea12c713b4df/volumes"
Oct 31 00:11:40.837967 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.837943    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42004e9-c4d2-435a-8f4b-0c283473b484" path="/var/lib/kubelet/pods/e42004e9-c4d2-435a-8f4b-0c283473b484/volumes"
Oct 31 00:11:40.926986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.926960    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"]
Oct 31 00:11:40.928849 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:11:40.928807    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a9e180_0d68_4a52_95fa_136ddc8bb425.slice/crio-44980f62cf16e6a98da009728bc1bda46f695a5dd6d35b64a3cb502c54f297f3 WatchSource:0}: Error finding container 44980f62cf16e6a98da009728bc1bda46f695a5dd6d35b64a3cb502c54f297f3: Status 404 returned error can't find the container with id 44980f62cf16e6a98da009728bc1bda46f695a5dd6d35b64a3cb502c54f297f3
Oct 31 00:11:40.930752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:40.930734    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:11:41.077565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:41.077535    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"]
Oct 31 00:11:41.078976 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:11:41.078950    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6f4213_ea07_410c_917d_127636230349.slice/crio-1834294ad18bc7eba0b4b3bf5568649045ebcd7e00281bac52ef47dfefd8133d WatchSource:0}: Error finding container 1834294ad18bc7eba0b4b3bf5568649045ebcd7e00281bac52ef47dfefd8133d: Status 404 returned error can't find the container with id 1834294ad18bc7eba0b4b3bf5568649045ebcd7e00281bac52ef47dfefd8133d
Oct 31 00:11:41.128252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:41.128214    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" event={"ID":"5e6f4213-ea07-410c-917d-127636230349","Type":"ContainerStarted","Data":"1834294ad18bc7eba0b4b3bf5568649045ebcd7e00281bac52ef47dfefd8133d"}
Oct 31 00:11:41.129510 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:41.129481    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" event={"ID":"d9a9e180-0d68-4a52-95fa-136ddc8bb425","Type":"ContainerStarted","Data":"44980f62cf16e6a98da009728bc1bda46f695a5dd6d35b64a3cb502c54f297f3"}
Oct 31 00:11:45.170860 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:45.170820    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" event={"ID":"5e6f4213-ea07-410c-917d-127636230349","Type":"ContainerStarted","Data":"d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d"}
Oct 31 00:11:45.171328 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:45.170892    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:45.172478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:45.172450    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" event={"ID":"d9a9e180-0d68-4a52-95fa-136ddc8bb425","Type":"ContainerStarted","Data":"41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea"}
Oct 31 00:11:45.172609 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:45.172533    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:45.285453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:45.285396    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" podStartSLOduration=1.850793061 podStartE2EDuration="5.285381488s" totalImagesPullingTime="3.434588427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:45.283031515 +0000 UTC m=+1810.573579835" watchObservedRunningTime="2026-10-31 00:11:45.285381488 +0000 UTC m=+1810.575929805"
Oct 31 00:11:45.395069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:45.395014    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" podStartSLOduration=1.716027875 podStartE2EDuration="5.394998928s" totalImagesPullingTime="3.678971053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:45.392502123 +0000 UTC m=+1810.683050443" watchObservedRunningTime="2026-10-31 00:11:45.394998928 +0000 UTC m=+1810.685547585"
Oct 31 00:11:54.799125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:54.799088    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7147-459f5f"]
Oct 31 00:11:54.803572 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:54.803540    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:11:54.856399 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:54.856362    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7147-459f5f"]
Oct 31 00:11:54.898574 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:54.898548    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bghhz\""
Oct 31 00:11:54.967611 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:54.967578    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdms\" (UniqueName: \"kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms\") pod \"test-curl-7147-459f5f\" (UID: \"66bec27c-8b28-46f0-9e36-82c9456aeb0b\") " pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:11:55.069464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:55.069371    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdms\" (UniqueName: \"kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms\") pod \"test-curl-7147-459f5f\" (UID: \"66bec27c-8b28-46f0-9e36-82c9456aeb0b\") " pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:11:55.127292 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:55.127262    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdms\" (UniqueName: \"kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms\") pod \"test-curl-7147-459f5f\" (UID: \"66bec27c-8b28-46f0-9e36-82c9456aeb0b\") " pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:11:55.415852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:55.415818    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:11:55.786858 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:55.786828    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7147-459f5f"]
Oct 31 00:11:55.788478 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:11:55.788453    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66bec27c_8b28_46f0_9e36_82c9456aeb0b.slice/crio-fe2b010a29907284b03b021bdfc3704bf7bdc39e456f1d14e1094688c3bac8e0 WatchSource:0}: Error finding container fe2b010a29907284b03b021bdfc3704bf7bdc39e456f1d14e1094688c3bac8e0: Status 404 returned error can't find the container with id fe2b010a29907284b03b021bdfc3704bf7bdc39e456f1d14e1094688c3bac8e0
Oct 31 00:11:56.182996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:56.182970    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:11:56.183400 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:56.183035    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:11:56.245208 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:11:56.245169    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7147-459f5f" event={"ID":"66bec27c-8b28-46f0-9e36-82c9456aeb0b","Type":"ContainerStarted","Data":"fe2b010a29907284b03b021bdfc3704bf7bdc39e456f1d14e1094688c3bac8e0"}
Oct 31 00:12:00.273298 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:00.273259    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7147-459f5f" event={"ID":"66bec27c-8b28-46f0-9e36-82c9456aeb0b","Type":"ContainerStarted","Data":"340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12"}
Oct 31 00:12:00.379930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:00.379856    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7147-459f5f" podStartSLOduration=2.341137591 podStartE2EDuration="6.379838898s" totalImagesPullingTime="4.038701307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:00.379410054 +0000 UTC m=+1825.669958378" watchObservedRunningTime="2026-10-31 00:12:00.379838898 +0000 UTC m=+1825.670387216"
Oct 31 00:12:01.909817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:01.909782    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7147-459f5f"]
Oct 31 00:12:01.914381 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:01.914351    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7147-459f5f"]
Oct 31 00:12:02.286540 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:02.286445    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7147-459f5f" podUID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" containerName="test-curl-7147-459f5f" containerID="cri-o://340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12" gracePeriod=2
Oct 31 00:12:02.334611 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:12:02.334568    4466 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7147-459f5f\" is forbidden: User \"system:node:ip-10-0-131-123.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-123.us-west-2.compute.internal' and this object" podUID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:12:04.525675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:04.525623    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:12:04.607477 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:04.607433    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms\" (UniqueName: \"kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms\") pod \"66bec27c-8b28-46f0-9e36-82c9456aeb0b\" (UID: \"66bec27c-8b28-46f0-9e36-82c9456aeb0b\") "
Oct 31 00:12:04.609764 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:04.609735    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms" pod "66bec27c-8b28-46f0-9e36-82c9456aeb0b" (UID: "66bec27c-8b28-46f0-9e36-82c9456aeb0b"). InnerVolumeSpecName "kube-api-access-9bdms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:04.709826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:04.709788    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bdms\" (UniqueName: \"kubernetes.io/projected/66bec27c-8b28-46f0-9e36-82c9456aeb0b-kube-api-access-9bdms\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:04.828046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:04.828015    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" path="/var/lib/kubelet/pods/66bec27c-8b28-46f0-9e36-82c9456aeb0b/volumes"
Oct 31 00:12:05.309509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:05.309476    4466 generic.go:356] "Generic (PLEG): container finished" podID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" containerID="340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12" exitCode=137
Oct 31 00:12:05.309737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:05.309518    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7147-459f5f"
Oct 31 00:12:05.309737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:05.309570    4466 scope.go:122] "RemoveContainer" containerID="340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12"
Oct 31 00:12:05.318815 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:05.318790    4466 scope.go:122] "RemoveContainer" containerID="340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12"
Oct 31 00:12:05.319079 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:12:05.319062    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12\": container with ID starting with 340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12 not found: ID does not exist" containerID="340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12"
Oct 31 00:12:05.319144 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:05.319088    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12"} err="failed to get container status \"340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12\": rpc error: code = NotFound desc = could not find container \"340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12\": container with ID starting with 340132d852bfe1a0b89b5f79489a384c84d9581ab50e538dc86cc88bbf745e12 not found: ID does not exist"
Oct 31 00:12:08.318266 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:12:08.318204    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:12:26.218236 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.218145    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"]
Oct 31 00:12:26.218788 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.218772    4466 cpu_manager.go:414] "containerMap: removing container" podUID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" containerName="test-curl-7147-459f5f"
Oct 31 00:12:26.218853 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.218789    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" containerName="test-curl-7147-459f5f"
Oct 31 00:12:26.218906 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.218897    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bec27c-8b28-46f0-9e36-82c9456aeb0b" containerName="test-curl-7147-459f5f"
Oct 31 00:12:26.227124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.227101    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.279496 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.279449    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"]
Oct 31 00:12:26.321065 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.321034    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-03c703-serving-cert\""
Oct 31 00:12:26.331974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.331938    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.332153 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.332127    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.332214 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.332171    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.332307 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.332291    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26fm\" (UniqueName: \"kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.433459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.433421    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.433692 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.433479    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.433692 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.433583    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26fm\" (UniqueName: \"kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.433692 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:12:26.433598    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-serving-cert: secret "maas-api-e2e-inf-a-03c703-serving-cert" not found
Oct 31 00:12:26.433692 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:12:26.433687    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls podName:e0f41986-e1cd-40b9-890b-e21f191c4736 nodeName:}" failed. No retries permitted until 2026-10-31 00:12:26.933669447 +0000 UTC m=+1852.224217767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls") pod "maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" (UID: "e0f41986-e1cd-40b9-890b-e21f191c4736") : secret "maas-api-e2e-inf-a-03c703-serving-cert" not found
Oct 31 00:12:26.433918 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.433802    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.434196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.434177    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.436262 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.436243    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.500412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.497983    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26fm\" (UniqueName: \"kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.942282 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.942227    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:26.944700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:26.944661    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\") pod \"maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:27.141153 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:27.141119    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:27.540460 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:27.540426    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"]
Oct 31 00:12:27.541457 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:12:27.541428    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f41986_e1cd_40b9_890b_e21f191c4736.slice/crio-64550a6e5b040237f744f3006d301dfebbd5c1821365f802a230aff2b6bb9641 WatchSource:0}: Error finding container 64550a6e5b040237f744f3006d301dfebbd5c1821365f802a230aff2b6bb9641: Status 404 returned error can't find the container with id 64550a6e5b040237f744f3006d301dfebbd5c1821365f802a230aff2b6bb9641
Oct 31 00:12:28.530071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:28.530034    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" event={"ID":"e0f41986-e1cd-40b9-890b-e21f191c4736","Type":"ContainerStarted","Data":"64550a6e5b040237f744f3006d301dfebbd5c1821365f802a230aff2b6bb9641"}
Oct 31 00:12:29.537710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:29.537669    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" event={"ID":"e0f41986-e1cd-40b9-890b-e21f191c4736","Type":"ContainerStarted","Data":"2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77"}
Oct 31 00:12:29.538123 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:29.537745    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:29.650112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:29.650063    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" podStartSLOduration=2.620864381 podStartE2EDuration="3.650047762s" totalImagesPullingTime="1.029183381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:29.648541364 +0000 UTC m=+1854.939089685" watchObservedRunningTime="2026-10-31 00:12:29.650047762 +0000 UTC m=+1854.940596132"
Oct 31 00:12:35.549714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:35.549619    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:12:38.317511 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:12:38.317450    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:12:40.565066 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.565033    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"]
Oct 31 00:12:40.574218 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.574141    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.665850 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.665791    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d9e782-openshift-default-dockercfg-rdj94\""
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685518    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685606    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685695    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685731    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4c5f\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685812    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685924    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.685947    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.686025    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.690658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.686104    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.762824 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.762784    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"]
Oct 31 00:12:40.807247 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807208    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807442 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807286    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807442 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807353    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807442 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807426    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807476    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4c5f\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807569    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807738    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.807815 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807768    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.808715 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.807853    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.808865 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.808747    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.809033 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.809012    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.809476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.809453    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.810348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.809716    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.810348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.810194    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.846681 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.843163    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.847888 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.842598    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.867873 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.867836    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.868887 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.868850    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4c5f\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f\") pod \"e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") " pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:40.889791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:40.889747    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:41.327174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.327144    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"]
Oct 31 00:12:41.328304 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:12:41.328279    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d9550a_a812_4abc_84ff_f4a62648aaf6.slice/crio-b5325ee24376d325ea9c74b38d4db6369c77a1b77c172f7b52d13ea2a6f74fdf WatchSource:0}: Error finding container b5325ee24376d325ea9c74b38d4db6369c77a1b77c172f7b52d13ea2a6f74fdf: Status 404 returned error can't find the container with id b5325ee24376d325ea9c74b38d4db6369c77a1b77c172f7b52d13ea2a6f74fdf
Oct 31 00:12:41.330918 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.330888    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:12:41.330994 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.330954    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:12:41.330994 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.330981    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:12:41.665135 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.665092    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" event={"ID":"38d9550a-a812-4abc-84ff-f4a62648aaf6","Type":"ContainerStarted","Data":"09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"}
Oct 31 00:12:41.665566 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.665168    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" event={"ID":"38d9550a-a812-4abc-84ff-f4a62648aaf6","Type":"ContainerStarted","Data":"b5325ee24376d325ea9c74b38d4db6369c77a1b77c172f7b52d13ea2a6f74fdf"}
Oct 31 00:12:41.784225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.784153    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podStartSLOduration=1.784134095 podStartE2EDuration="1.784134095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:41.779725365 +0000 UTC m=+1867.070273685" watchObservedRunningTime="2026-10-31 00:12:41.784134095 +0000 UTC m=+1867.074682416"
Oct 31 00:12:41.890180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:41.890149    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:42.896665 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:42.896604    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:43.676897 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:43.676862    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:43.677857 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:43.677838    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:12:48.947800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:48.947761    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"]
Oct 31 00:12:48.952339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:48.952314    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.077658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.076133    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d9e782-dockercfg-jcwqr\""
Oct 31 00:12:49.077658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.076244    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d9e782\""
Oct 31 00:12:49.080343 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.080310    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"]
Oct 31 00:12:49.138465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.138091    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vvj\" (UniqueName: \"kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj\") pod \"payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.139204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.139124    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.241398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.241303    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.241589 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.241571    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vvj\" (UniqueName: \"kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj\") pod \"payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.242180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.242150    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.300669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.300590    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vvj\" (UniqueName: \"kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj\") pod \"payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.570922 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.570828    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:49.943348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:49.943318    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"]
Oct 31 00:12:49.944264 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:12:49.944241    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6052d389_d76d_4728_85d2_0bbf266b5bc4.slice/crio-5888263fd8af4f6eb178822aa87ae90607278fc041f77fc403e765d8e3fd8882 WatchSource:0}: Error finding container 5888263fd8af4f6eb178822aa87ae90607278fc041f77fc403e765d8e3fd8882: Status 404 returned error can't find the container with id 5888263fd8af4f6eb178822aa87ae90607278fc041f77fc403e765d8e3fd8882
Oct 31 00:12:50.775058 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:50.775013    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v" event={"ID":"6052d389-d76d-4728-85d2-0bbf266b5bc4","Type":"ContainerStarted","Data":"019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b"}
Oct 31 00:12:50.775058 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:50.775065    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v" event={"ID":"6052d389-d76d-4728-85d2-0bbf266b5bc4","Type":"ContainerStarted","Data":"5888263fd8af4f6eb178822aa87ae90607278fc041f77fc403e765d8e3fd8882"}
Oct 31 00:12:50.775628 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:50.775098    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:12:50.886770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:12:50.886708    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v" podStartSLOduration=2.886693603 podStartE2EDuration="2.886693603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:50.885418453 +0000 UTC m=+1876.175966771" watchObservedRunningTime="2026-10-31 00:12:50.886693603 +0000 UTC m=+1876.177241921"
Oct 31 00:13:01.785150 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:01.785115    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:13:08.317931 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:13:08.317892    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:13:38.321299 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:13:38.321256    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:13:46.175761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.175726    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"]
Oct 31 00:13:46.180011 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.179987    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205409 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205375    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205564 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205438    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205564 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205508    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205646 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205597    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205646 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205628    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205775    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205879 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205866    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.205956 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.205942    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zrf\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.206042 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.206030    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.301425 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.301392    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"]
Oct 31 00:13:46.307122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307036    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307141    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307215    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307416 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307311    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zrf\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307416 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307399    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307532 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307454    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307584 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307528    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307678 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307565    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.307678 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.307673    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.308154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.308125    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.308281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.308262    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.308693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.308661    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.310588 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.308934    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.311388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.311247    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.314074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.312742    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-348a33-openshift-default-dockercfg-545pr\""
Oct 31 00:13:46.315776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.315754    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.315937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.315917    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.384762 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.384729    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.392362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.392321    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zrf\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf\") pod \"e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") " pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.494550 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.494463    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:46.917472 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.917435    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"]
Oct 31 00:13:46.918231 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:13:46.918206    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9eafdaa_1fb5_4ca0_a3b4_5c168464fac4.slice/crio-98755c0957c73561199dd74de555494cfd82aa3357002e0938dbbcd1006e9112 WatchSource:0}: Error finding container 98755c0957c73561199dd74de555494cfd82aa3357002e0938dbbcd1006e9112: Status 404 returned error can't find the container with id 98755c0957c73561199dd74de555494cfd82aa3357002e0938dbbcd1006e9112
Oct 31 00:13:46.920889 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.920857    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:13:46.920952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.920924    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:13:46.920985 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:46.920951    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:13:47.206114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:47.206014    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" event={"ID":"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4","Type":"ContainerStarted","Data":"9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"}
Oct 31 00:13:47.206114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:47.206067    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" event={"ID":"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4","Type":"ContainerStarted","Data":"98755c0957c73561199dd74de555494cfd82aa3357002e0938dbbcd1006e9112"}
Oct 31 00:13:47.323964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:47.323426    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podStartSLOduration=1.323407349 podStartE2EDuration="1.323407349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:47.319729547 +0000 UTC m=+1932.610277860" watchObservedRunningTime="2026-10-31 00:13:47.323407349 +0000 UTC m=+1932.613955669"
Oct 31 00:13:47.495388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:47.495299    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:47.501765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:47.501693    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:48.213109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:48.213077    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:48.214885 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:48.214856    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:13:55.520217 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.520179    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"]
Oct 31 00:13:55.524527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.524507    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.623309 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.623269    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-348a33-dockercfg-d57sz\""
Oct 31 00:13:55.623528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.623442    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-348a33\""
Oct 31 00:13:55.629217 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.629180    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"]
Oct 31 00:13:55.669936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.667425    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.669936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.667500    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhr8c\" (UniqueName: \"kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c\") pod \"payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.774941 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.774168    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.774941 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.774259    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhr8c\" (UniqueName: \"kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c\") pod \"payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.776431 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.776341    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.833785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.833754    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhr8c\" (UniqueName: \"kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c\") pod \"payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:55.842426 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:55.842389    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:56.221911 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:56.221880    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"]
Oct 31 00:13:56.222765 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:13:56.222737    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa15182f_8783_49ce_a4e2_a32de14f12e5.slice/crio-e1e7d35035a5cd6254e8b3c1f97caab91b21043732dea2f3d6d39db39ed5999c WatchSource:0}: Error finding container e1e7d35035a5cd6254e8b3c1f97caab91b21043732dea2f3d6d39db39ed5999c: Status 404 returned error can't find the container with id e1e7d35035a5cd6254e8b3c1f97caab91b21043732dea2f3d6d39db39ed5999c
Oct 31 00:13:56.277395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:56.277368    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2" event={"ID":"fa15182f-8783-49ce-a4e2-a32de14f12e5","Type":"ContainerStarted","Data":"e1e7d35035a5cd6254e8b3c1f97caab91b21043732dea2f3d6d39db39ed5999c"}
Oct 31 00:13:57.285164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:57.285132    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2" event={"ID":"fa15182f-8783-49ce-a4e2-a32de14f12e5","Type":"ContainerStarted","Data":"5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1"}
Oct 31 00:13:57.287434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:57.285214    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:13:57.396409 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:13:57.396348    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2" podStartSLOduration=2.396333346 podStartE2EDuration="2.396333346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:57.39444931 +0000 UTC m=+1942.684997633" watchObservedRunningTime="2026-10-31 00:13:57.396333346 +0000 UTC m=+1942.686881663"
Oct 31 00:14:07.024072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.024032    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"]
Oct 31 00:14:07.028322 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.028299    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.090412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.090375    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"]
Oct 31 00:14:07.118263 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.118209    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.118520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.118503    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlbw\" (UniqueName: \"kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.125431 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.125399    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a251fdfb-dockercfg-nmpq4\""
Oct 31 00:14:07.125431 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.125422    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a251fdfb\""
Oct 31 00:14:07.221612 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.221557    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.233960 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.223486    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.234164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.234006    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlbw\" (UniqueName: \"kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.292073 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.291997    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlbw\" (UniqueName: \"kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw\") pod \"payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") " pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.343167 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.343127    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:07.721707 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:07.721672    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"]
Oct 31 00:14:07.722418 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:14:07.722392    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47abc60_9568_43e1_82c7_e937af2ce01f.slice/crio-5a4b6a0a8a4c1b7296d98c57784e0c150ca5d984aaaf773c6e042580a34f117f WatchSource:0}: Error finding container 5a4b6a0a8a4c1b7296d98c57784e0c150ca5d984aaaf773c6e042580a34f117f: Status 404 returned error can't find the container with id 5a4b6a0a8a4c1b7296d98c57784e0c150ca5d984aaaf773c6e042580a34f117f
Oct 31 00:14:08.298942 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:08.298913    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:14:08.318214 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:14:08.318167    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:14:08.356457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:08.356424    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v" event={"ID":"a47abc60-9568-43e1-82c7-e937af2ce01f","Type":"ContainerStarted","Data":"47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6"}
Oct 31 00:14:08.356669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:08.356480    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v" event={"ID":"a47abc60-9568-43e1-82c7-e937af2ce01f","Type":"ContainerStarted","Data":"5a4b6a0a8a4c1b7296d98c57784e0c150ca5d984aaaf773c6e042580a34f117f"}
Oct 31 00:14:08.356669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:08.356513    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:08.529146 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:08.523237    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v" podStartSLOduration=2.523216278 podStartE2EDuration="2.523216278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:08.520226338 +0000 UTC m=+1953.810774655" watchObservedRunningTime="2026-10-31 00:14:08.523216278 +0000 UTC m=+1953.813764598"
Oct 31 00:14:11.454483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.454436    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"]
Oct 31 00:14:11.460162 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.460116    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:11.514008 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.513786    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-460d4e-dockercfg-wrxbk\""
Oct 31 00:14:11.563332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.563285    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-460d4e\""
Oct 31 00:14:11.568334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.568298    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"]
Oct 31 00:14:11.646684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.643899    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:11.646684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.643962    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z6b\" (UniqueName: \"kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b\") pod \"payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:11.747902 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.747796    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:11.747902 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.747875    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z6b\" (UniqueName: \"kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b\") pod \"payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:11.749559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.749526    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:11.806332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:11.806301    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z6b\" (UniqueName: \"kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b\") pod \"payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:12.078844 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:12.078762    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:12.452348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:12.452315    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"]
Oct 31 00:14:12.453336 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:14:12.453309    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb0b23f_693f_4bd2_af57_e416351adf33.slice/crio-fef974751d4b0982464de91068924d166f24851ab0e4a7d3045b7c39aedef5f5 WatchSource:0}: Error finding container fef974751d4b0982464de91068924d166f24851ab0e4a7d3045b7c39aedef5f5: Status 404 returned error can't find the container with id fef974751d4b0982464de91068924d166f24851ab0e4a7d3045b7c39aedef5f5
Oct 31 00:14:13.431418 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:13.430897    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj" event={"ID":"fbb0b23f-693f-4bd2-af57-e416351adf33","Type":"ContainerStarted","Data":"60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402"}
Oct 31 00:14:13.431418 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:13.430965    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj" event={"ID":"fbb0b23f-693f-4bd2-af57-e416351adf33","Type":"ContainerStarted","Data":"fef974751d4b0982464de91068924d166f24851ab0e4a7d3045b7c39aedef5f5"}
Oct 31 00:14:13.432037 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:13.432006    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:13.544215 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:13.544154    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj" podStartSLOduration=2.544134423 podStartE2EDuration="2.544134423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:13.54066494 +0000 UTC m=+1958.831213265" watchObservedRunningTime="2026-10-31 00:14:13.544134423 +0000 UTC m=+1958.834682740"
Oct 31 00:14:19.366074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:19.366042    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:14:19.986060 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:19.986026    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"]
Oct 31 00:14:19.986318 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:19.986294    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" containerID="cri-o://f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c" gracePeriod=30
Oct 31 00:14:21.063269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:21.063224    4466 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:14:21.063714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:21.063304    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:14:25.272068 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.272000    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 31 00:14:25.378091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378056    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378096    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378124    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378203    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378315    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378430    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378439    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:25.378516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378483    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:25.378691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378525    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378623    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378679    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy\") pod \"7147a543-9acb-4d97-b8cd-1f95da39b70b\" (UID: \"7147a543-9acb-4d97-b8cd-1f95da39b70b\") "
Oct 31 00:14:25.378834 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378683    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:25.378882 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378841    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:25.378928 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.378887    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:25.379325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.379301    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.379325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.379325    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.379508 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.379335    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7147a543-9acb-4d97-b8cd-1f95da39b70b-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.379508 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.379344    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.379508 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.379354    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.380585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.380553    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "kube-api-access-hv48g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:25.380893 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.380873    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:25.380957 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.380879    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:25.381000 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.380950    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy" pod "7147a543-9acb-4d97-b8cd-1f95da39b70b" (UID: "7147a543-9acb-4d97-b8cd-1f95da39b70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:25.448032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.447999    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:14:25.480956 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.480919    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.480956 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.480958    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.481159 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.480975    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv48g\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-kube-api-access-hv48g\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.481159 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.480991    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7147a543-9acb-4d97-b8cd-1f95da39b70b-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:25.532469 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.532389    4466 generic.go:356] "Generic (PLEG): container finished" podID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerID="f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c" exitCode=0
Oct 31 00:14:25.532469 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.532425    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" event={"ID":"7147a543-9acb-4d97-b8cd-1f95da39b70b","Type":"ContainerDied","Data":"f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c"}
Oct 31 00:14:25.532469 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.532439    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"
Oct 31 00:14:25.532469 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.532462    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm" event={"ID":"7147a543-9acb-4d97-b8cd-1f95da39b70b","Type":"ContainerDied","Data":"9b2eeb205334f4075f03271c9c651f8ea4cb2f4e95e1fad2e38e7fea135bff5f"}
Oct 31 00:14:25.532780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.532484    4466 scope.go:122] "RemoveContainer" containerID="f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c"
Oct 31 00:14:25.543348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.543331    4466 scope.go:122] "RemoveContainer" containerID="f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c"
Oct 31 00:14:25.546301 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:14:25.546268    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c\": container with ID starting with f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c not found: ID does not exist" containerID="f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c"
Oct 31 00:14:25.546412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.546309    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c"} err="failed to get container status \"f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c\": rpc error: code = NotFound desc = could not find container \"f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c\": container with ID starting with f2e182a0630434d9ddfbc71b9605b55d66867ad1e9a02e964bbf82fee682688c not found: ID does not exist"
Oct 31 00:14:25.670131 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.670084    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"]
Oct 31 00:14:25.686091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:25.686056    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-txzbm"]
Oct 31 00:14:26.828505 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:26.828468    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" path="/var/lib/kubelet/pods/7147a543-9acb-4d97-b8cd-1f95da39b70b/volumes"
Oct 31 00:14:29.849320 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.849285    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"]
Oct 31 00:14:29.850126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.850104    4466 cpu_manager.go:414] "containerMap: removing container" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy"
Oct 31 00:14:29.850241 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.850127    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy"
Oct 31 00:14:29.850332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.850319    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7147a543-9acb-4d97-b8cd-1f95da39b70b" containerName="istio-proxy"
Oct 31 00:14:29.853803 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.853783    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:29.911208 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.911173    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c9c21e\""
Oct 31 00:14:29.963502 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.963454    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:29.966975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.966925    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz5b\" (UniqueName: \"kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b\") pod \"payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:29.976871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.963989    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c9c21e-dockercfg-qlftb\""
Oct 31 00:14:29.983778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:29.983744    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"]
Oct 31 00:14:30.042097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.042063    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"]
Oct 31 00:14:30.047537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.046971    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.079658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.079395    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz5b\" (UniqueName: \"kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b\") pod \"payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:30.079658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.079659    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:30.080527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.080498    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:30.086151 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.086117    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"]
Oct 31 00:14:30.171362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.171280    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz5b\" (UniqueName: \"kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b\") pod \"payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:30.171558 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.171541    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:30.181760 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.181724    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22csk\" (UniqueName: \"kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk\") pod \"payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.182097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.182032    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.194346 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.194317    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"]
Oct 31 00:14:30.196618 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.196334    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy" containerID="cri-o://21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800" gracePeriod=30
Oct 31 00:14:30.298935 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.295941    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.298935 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.296095    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22csk\" (UniqueName: \"kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk\") pod \"payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.298935 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.297436    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.355262 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.355227    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22csk\" (UniqueName: \"kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk\") pod \"payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.365749 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.365686    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:30.606736 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.606699    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"]
Oct 31 00:14:30.608169 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:14:30.608142    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c24278_cb5d_4762_8841_5c0fcf8127ad.slice/crio-d5f080276754e2b0d7921be73810c38c0fbf9b1277a3a03d4a701d48d08a2e51 WatchSource:0}: Error finding container d5f080276754e2b0d7921be73810c38c0fbf9b1277a3a03d4a701d48d08a2e51: Status 404 returned error can't find the container with id d5f080276754e2b0d7921be73810c38c0fbf9b1277a3a03d4a701d48d08a2e51
Oct 31 00:14:30.771167 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:30.771134    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"]
Oct 31 00:14:30.771744 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:14:30.771712    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02028cc9_2ea9_437b_9d75_b6221a81ce43.slice/crio-bb0b4f2f118554877e6ea0527ab31257981e32b256892de98cd2ffb69412790d WatchSource:0}: Error finding container bb0b4f2f118554877e6ea0527ab31257981e32b256892de98cd2ffb69412790d: Status 404 returned error can't find the container with id bb0b4f2f118554877e6ea0527ab31257981e32b256892de98cd2ffb69412790d
Oct 31 00:14:31.589405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.589364    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss" event={"ID":"02028cc9-2ea9-437b-9d75-b6221a81ce43","Type":"ContainerStarted","Data":"2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff"}
Oct 31 00:14:31.589902 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.589428    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss" event={"ID":"02028cc9-2ea9-437b-9d75-b6221a81ce43","Type":"ContainerStarted","Data":"bb0b4f2f118554877e6ea0527ab31257981e32b256892de98cd2ffb69412790d"}
Oct 31 00:14:31.589902 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.589592    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:31.591153 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.591120    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf" event={"ID":"78c24278-cb5d-4762-8841-5c0fcf8127ad","Type":"ContainerStarted","Data":"e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f"}
Oct 31 00:14:31.591274 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.591168    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf" event={"ID":"78c24278-cb5d-4762-8841-5c0fcf8127ad","Type":"ContainerStarted","Data":"d5f080276754e2b0d7921be73810c38c0fbf9b1277a3a03d4a701d48d08a2e51"}
Oct 31 00:14:31.591274 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.591194    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:31.700025 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.699976    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss" podStartSLOduration=2.699962689 podStartE2EDuration="2.699962689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:31.69961068 +0000 UTC m=+1976.990158998" watchObservedRunningTime="2026-10-31 00:14:31.699962689 +0000 UTC m=+1976.990511006"
Oct 31 00:14:31.810477 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:31.810422    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf" podStartSLOduration=2.8104043340000002 podStartE2EDuration="2.810404334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:31.809431863 +0000 UTC m=+1977.099980182" watchObservedRunningTime="2026-10-31 00:14:31.810404334 +0000 UTC m=+1977.100952653"
Oct 31 00:14:35.481296 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.481233    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:14:35.573503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573467    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.573726 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573697    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.573809 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573756    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.573865 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573843    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.573915 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573906    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.573988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573962    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.574026 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.573993    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.574069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574050    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.574116 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574070    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.574182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574159    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.574247 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574231    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert\") pod \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\" (UID: \"5f0cab75-97d3-444b-91cf-a53286aa4ccc\") "
Oct 31 00:14:35.574523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574498    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.574523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574474    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.574724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574619    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:35.575130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574937    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.575130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574960    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.575130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574974    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.575130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.574988    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.575130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.575001    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.576323 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.576297    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.576436 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.576336    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "kube-api-access-v55bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:35.576626 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.576604    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:35.576790 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.576766    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo" pod "5f0cab75-97d3-444b-91cf-a53286aa4ccc" (UID: "5f0cab75-97d3-444b-91cf-a53286aa4ccc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:35.618929 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.618891    4466 generic.go:356] "Generic (PLEG): container finished" podID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerID="21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800" exitCode=0
Oct 31 00:14:35.619125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.618956    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"
Oct 31 00:14:35.619125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.618971    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" event={"ID":"5f0cab75-97d3-444b-91cf-a53286aa4ccc","Type":"ContainerDied","Data":"21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800"}
Oct 31 00:14:35.619125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.619075    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh" event={"ID":"5f0cab75-97d3-444b-91cf-a53286aa4ccc","Type":"ContainerDied","Data":"02f3e1e7b7b1bdf80d6e37d7b05b1b95ec5fa20d48c9cd1d0cd52ec68265f7b9"}
Oct 31 00:14:35.619125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.619107    4466 scope.go:122] "RemoveContainer" containerID="21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800"
Oct 31 00:14:35.630814 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.630786    4466 scope.go:122] "RemoveContainer" containerID="21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800"
Oct 31 00:14:35.631102 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:14:35.631083    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800\": container with ID starting with 21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800 not found: ID does not exist" containerID="21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800"
Oct 31 00:14:35.631175 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.631112    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800"} err="failed to get container status \"21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800\": rpc error: code = NotFound desc = could not find container \"21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800\": container with ID starting with 21f72dfb10d26e23e77c834da22bab7772aa6e38f88c3c8f368fcb66c44ac800 not found: ID does not exist"
Oct 31 00:14:35.676742 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.676705    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55bd\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-kube-api-access-v55bd\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.676742 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.676744    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.676956 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.676765    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.676956 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.676778    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0cab75-97d3-444b-91cf-a53286aa4ccc-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.750309 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.750199    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"]
Oct 31 00:14:35.755100 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:35.755070    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-smvsh"]
Oct 31 00:14:36.244625 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.244588    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"]
Oct 31 00:14:36.244836 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.244807    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" containerID="cri-o://975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef" gracePeriod=30
Oct 31 00:14:36.377019 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.376983    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"]
Oct 31 00:14:36.377272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.377242    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" containerID="cri-o://8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0" gracePeriod=30
Oct 31 00:14:36.412737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.412684    4466 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:14:36.412916 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.412766    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:14:36.829046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:36.829016    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" path="/var/lib/kubelet/pods/5f0cab75-97d3-444b-91cf-a53286aa4ccc/volumes"
Oct 31 00:14:38.179204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:38.179168    4466 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:14:38.179645 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:38.179231    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:14:38.317899 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:14:38.317862    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:14:41.521232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.521162    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:14:41.650238 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.650204    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 31 00:14:41.660197 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.660168    4466 generic.go:356] "Generic (PLEG): container finished" podID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerID="975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef" exitCode=0
Oct 31 00:14:41.660360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.660218    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"
Oct 31 00:14:41.660360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.660244    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" event={"ID":"34f65b7e-e40c-4a56-afcc-f23f00778bf6","Type":"ContainerDied","Data":"975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef"}
Oct 31 00:14:41.660360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.660302    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt" event={"ID":"34f65b7e-e40c-4a56-afcc-f23f00778bf6","Type":"ContainerDied","Data":"031831003ddcde98775e122993d671d40c230b43ad634b9ec4bd305c5bbd60da"}
Oct 31 00:14:41.660360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.660330    4466 scope.go:122] "RemoveContainer" containerID="975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef"
Oct 31 00:14:41.662127 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.662103    4466 generic.go:356] "Generic (PLEG): container finished" podID="8bcee6fe-3f93-499a-aca5-406342361197" containerID="8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0" exitCode=0
Oct 31 00:14:41.662268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.662153    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"
Oct 31 00:14:41.662268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.662173    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" event={"ID":"8bcee6fe-3f93-499a-aca5-406342361197","Type":"ContainerDied","Data":"8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0"}
Oct 31 00:14:41.662268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.662225    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8" event={"ID":"8bcee6fe-3f93-499a-aca5-406342361197","Type":"ContainerDied","Data":"35f9b487fba6a809b5adcdaf571577fabafe63fcef86f7fa0711b6b6f4b47221"}
Oct 31 00:14:41.667153 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667105    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667283 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667256    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667352 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667293    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667352 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667331    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667458 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667410    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667513 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667497    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667609 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667590    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667725 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667706    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.667812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.667795    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy\") pod \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\" (UID: \"34f65b7e-e40c-4a56-afcc-f23f00778bf6\") "
Oct 31 00:14:41.671148 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.671115    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:41.671900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.671871    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.672111 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.672086    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.672297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.672273    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.672507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.672460    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.672728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.672706    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.674112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.674069    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "kube-api-access-l2lpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.677578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.677556    4466 scope.go:122] "RemoveContainer" containerID="975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef"
Oct 31 00:14:41.678321 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:14:41.677960    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef\": container with ID starting with 975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef not found: ID does not exist" containerID="975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef"
Oct 31 00:14:41.678321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.677999    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef"} err="failed to get container status \"975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef\": rpc error: code = NotFound desc = could not find container \"975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef\": container with ID starting with 975159feb8db6858da182ad70d60a36ca75aa1fae0e520848d562af7645dfeef not found: ID does not exist"
Oct 31 00:14:41.678321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.678025    4466 scope.go:122] "RemoveContainer" containerID="8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0"
Oct 31 00:14:41.678321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.678077    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:41.678321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.678210    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token" pod "34f65b7e-e40c-4a56-afcc-f23f00778bf6" (UID: "34f65b7e-e40c-4a56-afcc-f23f00778bf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.691941 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.691921    4466 scope.go:122] "RemoveContainer" containerID="8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0"
Oct 31 00:14:41.692284 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:14:41.692261    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0\": container with ID starting with 8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0 not found: ID does not exist" containerID="8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0"
Oct 31 00:14:41.692364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.692298    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0"} err="failed to get container status \"8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0\": rpc error: code = NotFound desc = could not find container \"8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0\": container with ID starting with 8f343902eb2f7c0a65213dde50fa5144375c3f33d5c8ad1143fdd1ba3a15edb0 not found: ID does not exist"
Oct 31 00:14:41.770019 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.769987    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770081    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770132    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770169    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770195    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770244    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770292    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.770465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770323    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770355    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770505    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert\") pod \"8bcee6fe-3f93-499a-aca5-406342361197\" (UID: \"8bcee6fe-3f93-499a-aca5-406342361197\") "
Oct 31 00:14:41.770713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770554    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.770884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770860    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.771007 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770961    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.770801    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771320    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771341    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771353    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771362    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771370    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771379    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcee6fe-3f93-499a-aca5-406342361197-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771387    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771396    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771406    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771419    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2lpj\" (UniqueName: \"kubernetes.io/projected/34f65b7e-e40c-4a56-afcc-f23f00778bf6-kube-api-access-l2lpj\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771433    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f65b7e-e40c-4a56-afcc-f23f00778bf6-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771441    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.771940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.771451    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f65b7e-e40c-4a56-afcc-f23f00778bf6-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.772955 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.772927    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "kube-api-access-bkrm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.773049 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.772973    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.773049 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.772988    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:41.773049 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.773030    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token" pod "8bcee6fe-3f93-499a-aca5-406342361197" (UID: "8bcee6fe-3f93-499a-aca5-406342361197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.873253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.873212    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.873253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.873245    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.873253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.873260    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrm2\" (UniqueName: \"kubernetes.io/projected/8bcee6fe-3f93-499a-aca5-406342361197-kube-api-access-bkrm2\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.873499 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.873273    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcee6fe-3f93-499a-aca5-406342361197-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.873499 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:41.873286    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcee6fe-3f93-499a-aca5-406342361197-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:42.088173 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.088140    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"]
Oct 31 00:14:42.091353 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.091328    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-t5hkt"]
Oct 31 00:14:42.210656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.210609    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"]
Oct 31 00:14:42.218168 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.218127    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-q6bx8"]
Oct 31 00:14:42.600266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.600237    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:14:42.600756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.600314    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:14:42.835903 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.835865    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" path="/var/lib/kubelet/pods/34f65b7e-e40c-4a56-afcc-f23f00778bf6/volumes"
Oct 31 00:14:42.836607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:14:42.836586    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcee6fe-3f93-499a-aca5-406342361197" path="/var/lib/kubelet/pods/8bcee6fe-3f93-499a-aca5-406342361197/volumes"
Oct 31 00:15:00.216598 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.216565    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"]
Oct 31 00:15:00.217459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217438    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy"
Oct 31 00:15:00.217459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217460    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy"
Oct 31 00:15:00.217623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217527    4466 cpu_manager.go:414] "containerMap: removing container" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy"
Oct 31 00:15:00.217623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217536    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy"
Oct 31 00:15:00.217623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217552    4466 cpu_manager.go:414] "containerMap: removing container" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy"
Oct 31 00:15:00.217623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217560    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy"
Oct 31 00:15:00.217838 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217727    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcee6fe-3f93-499a-aca5-406342361197" containerName="istio-proxy"
Oct 31 00:15:00.217838 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217741    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f65b7e-e40c-4a56-afcc-f23f00778bf6" containerName="istio-proxy"
Oct 31 00:15:00.217838 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.217753    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0cab75-97d3-444b-91cf-a53286aa4ccc" containerName="istio-proxy"
Oct 31 00:15:00.221338 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.221317    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"]
Oct 31 00:15:00.221456 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.221435    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.225418 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.225395    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.255983 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.255949    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"]
Oct 31 00:15:00.260321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.260301    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.276151 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.276123    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"]
Oct 31 00:15:00.280170 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.280147    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.304149 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.304113    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"]
Oct 31 00:15:00.309442 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.309410    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.320736 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.319551    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:15:00.325193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.325166    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"]
Oct 31 00:15:00.330352 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.330319    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"]
Oct 31 00:15:00.359263 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.359196    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"]
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.360740    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.360929    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.361035    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.361088    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.361271    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.361495    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbth9\" (UniqueName: \"kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.362429    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg24k\" (UniqueName: \"kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.362502    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6tf\" (UniqueName: \"kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf\") pod \"maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.362570    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hf2\" (UniqueName: \"kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.365016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.362605    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spl7\" (UniqueName: \"kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.398101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.398060    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"]
Oct 31 00:15:00.448659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.446906    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"]
Oct 31 00:15:00.464615 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.464577    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.464967 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.464943    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.465144 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.465126    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.465266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.465252    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.465398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.465384    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.465485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.465460    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.465626 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.465611    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbth9\" (UniqueName: \"kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.465871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.465848    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg24k\" (UniqueName: \"kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.466014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.466000    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6tf\" (UniqueName: \"kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf\") pod \"maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.466151 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.466137    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hf2\" (UniqueName: \"kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.466292 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.466277    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spl7\" (UniqueName: \"kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.466477 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.466450    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.466547 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.466180    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.466547 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.466320    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.467327 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.467275    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.541712 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.529020    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbth9\" (UniqueName: \"kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9\") pod \"maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.558079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.557504    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg24k\" (UniqueName: \"kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.558279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.558178    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:00.560186 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.560157    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6tf\" (UniqueName: \"kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf\") pod \"maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.560366 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.560280    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spl7\" (UniqueName: \"kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7\") pod \"maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.560546 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.560526    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hf2\" (UniqueName: \"kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.574646 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.574608    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:00.597855 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.597822    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:00.645884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.645674    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:00.861679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.852917    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:00.965415 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.965367    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"]
Oct 31 00:15:00.991574 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:15:00.991351    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bce1b4_77cd_4f07_b8c7_28b0a2ac1bcb.slice/crio-e8a3c9b77191a2826c711497923b57a14b521fcb636528aeb48b666789e2d7a4 WatchSource:0}: Error finding container e8a3c9b77191a2826c711497923b57a14b521fcb636528aeb48b666789e2d7a4: Status 404 returned error can't find the container with id e8a3c9b77191a2826c711497923b57a14b521fcb636528aeb48b666789e2d7a4
Oct 31 00:15:00.992410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:00.992385    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"]
Oct 31 00:15:01.076180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.076148    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"]
Oct 31 00:15:01.077105 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:15:01.077080    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736810f3_f4c6_4671_be23_dde23a413f83.slice/crio-5b815e31215269173499945739a9868c02a195c91bdb6b41d359674925d085d9 WatchSource:0}: Error finding container 5b815e31215269173499945739a9868c02a195c91bdb6b41d359674925d085d9: Status 404 returned error can't find the container with id 5b815e31215269173499945739a9868c02a195c91bdb6b41d359674925d085d9
Oct 31 00:15:01.083788 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.083763    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"]
Oct 31 00:15:01.084867 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:15:01.084844    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56611d6_ed45_40fa_ace0_6f1616c1f0ca.slice/crio-8948dccfc3ac9f959d467abd27932df2a8a65dc8200bc15a2d0cd5645e5036a9 WatchSource:0}: Error finding container 8948dccfc3ac9f959d467abd27932df2a8a65dc8200bc15a2d0cd5645e5036a9: Status 404 returned error can't find the container with id 8948dccfc3ac9f959d467abd27932df2a8a65dc8200bc15a2d0cd5645e5036a9
Oct 31 00:15:01.259928 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.259839    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"]
Oct 31 00:15:01.264160 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:15:01.264131    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d40534_5cef_4780_9d12_39785abe6b05.slice/crio-ab3431790bfa8f261fd7f4c10aee8ca668148c8615149789c51d7b1e8303917e WatchSource:0}: Error finding container ab3431790bfa8f261fd7f4c10aee8ca668148c8615149789c51d7b1e8303917e: Status 404 returned error can't find the container with id ab3431790bfa8f261fd7f4c10aee8ca668148c8615149789c51d7b1e8303917e
Oct 31 00:15:01.804511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.804473    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" event={"ID":"b4d40534-5cef-4780-9d12-39785abe6b05","Type":"ContainerStarted","Data":"ab3431790bfa8f261fd7f4c10aee8ca668148c8615149789c51d7b1e8303917e"}
Oct 31 00:15:01.805848 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.805821    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerStarted","Data":"e8a3c9b77191a2826c711497923b57a14b521fcb636528aeb48b666789e2d7a4"}
Oct 31 00:15:01.807075 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.807050    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerStarted","Data":"8948dccfc3ac9f959d467abd27932df2a8a65dc8200bc15a2d0cd5645e5036a9"}
Oct 31 00:15:01.808344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.808320    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" event={"ID":"736810f3-f4c6-4671-be23-dde23a413f83","Type":"ContainerStarted","Data":"5b815e31215269173499945739a9868c02a195c91bdb6b41d359674925d085d9"}
Oct 31 00:15:01.809502 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:01.809479    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerStarted","Data":"dbfb51d1b13fd94b9817d574a227284f2eaa35b19b897455ab9d1f8a4b42869a"}
Oct 31 00:15:07.935435 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:07.935350    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerStarted","Data":"6939a12f6000ac6fb9939e9d9e47d3d453766fceda6352e57dcd462c6bba64ee"}
Oct 31 00:15:07.938142 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:07.938113    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerStarted","Data":"1f5d3de1d2f92697dcb00826714033502d04897dba69df9ab1ce23f973c50008"}
Oct 31 00:15:08.050464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.050399    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" podStartSLOduration=1.775722385 podStartE2EDuration="8.050380616s" totalImagesPullingTime="6.274658231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:08.048731884 +0000 UTC m=+2013.339280205" watchObservedRunningTime="2026-10-31 00:15:08.050380616 +0000 UTC m=+2013.340928934"
Oct 31 00:15:08.160822 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.160757    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" podStartSLOduration=2.02326144 podStartE2EDuration="8.160735287s" totalImagesPullingTime="6.137473847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:08.158836602 +0000 UTC m=+2013.449384922" watchObservedRunningTime="2026-10-31 00:15:08.160735287 +0000 UTC m=+2013.451283606"
Oct 31 00:15:08.319476 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:08.319416    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:15:08.947912 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.947881    4466 generic.go:356] "Generic (PLEG): container finished" podID="736810f3-f4c6-4671-be23-dde23a413f83" containerID="80ee78a9ac368ea5c9d8da8e0bf0c879812a2641346ed8c5964dc314eb0bba6c" exitCode=60
Oct 31 00:15:08.948276 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.947968    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" event={"ID":"736810f3-f4c6-4671-be23-dde23a413f83","Type":"ContainerDied","Data":"80ee78a9ac368ea5c9d8da8e0bf0c879812a2641346ed8c5964dc314eb0bba6c"}
Oct 31 00:15:08.948276 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.948258    4466 scope.go:122] "RemoveContainer" containerID="80ee78a9ac368ea5c9d8da8e0bf0c879812a2641346ed8c5964dc314eb0bba6c"
Oct 31 00:15:08.950056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.949932    4466 generic.go:356] "Generic (PLEG): container finished" podID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerID="1f5d3de1d2f92697dcb00826714033502d04897dba69df9ab1ce23f973c50008" exitCode=60
Oct 31 00:15:08.950056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.949958    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerDied","Data":"1f5d3de1d2f92697dcb00826714033502d04897dba69df9ab1ce23f973c50008"}
Oct 31 00:15:08.950293 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.950278    4466 scope.go:122] "RemoveContainer" containerID="1f5d3de1d2f92697dcb00826714033502d04897dba69df9ab1ce23f973c50008"
Oct 31 00:15:08.951977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.951954    4466 generic.go:356] "Generic (PLEG): container finished" podID="b4d40534-5cef-4780-9d12-39785abe6b05" containerID="9fa9be89037fd062c63c5b4c6347e238f7952a05c38087ae53b1e3fb21bba31d" exitCode=60
Oct 31 00:15:08.952107 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.952034    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" event={"ID":"b4d40534-5cef-4780-9d12-39785abe6b05","Type":"ContainerDied","Data":"9fa9be89037fd062c63c5b4c6347e238f7952a05c38087ae53b1e3fb21bba31d"}
Oct 31 00:15:08.952233 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.952216    4466 scope.go:122] "RemoveContainer" containerID="9fa9be89037fd062c63c5b4c6347e238f7952a05c38087ae53b1e3fb21bba31d"
Oct 31 00:15:08.953806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.953784    4466 generic.go:356] "Generic (PLEG): container finished" podID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerID="6939a12f6000ac6fb9939e9d9e47d3d453766fceda6352e57dcd462c6bba64ee" exitCode=60
Oct 31 00:15:08.953990 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.953863    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerDied","Data":"6939a12f6000ac6fb9939e9d9e47d3d453766fceda6352e57dcd462c6bba64ee"}
Oct 31 00:15:08.954226 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.954198    4466 scope.go:122] "RemoveContainer" containerID="6939a12f6000ac6fb9939e9d9e47d3d453766fceda6352e57dcd462c6bba64ee"
Oct 31 00:15:08.956206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.956186    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerStarted","Data":"94c9f0e8d8230ebcc54c34b7f23f4c1e2b307f9b54fac318bfb002566b1b6af4"}
Oct 31 00:15:08.956390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:08.956374    4466 scope.go:122] "RemoveContainer" containerID="94c9f0e8d8230ebcc54c34b7f23f4c1e2b307f9b54fac318bfb002566b1b6af4"
Oct 31 00:15:09.967048 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.967011    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" event={"ID":"736810f3-f4c6-4671-be23-dde23a413f83","Type":"ContainerStarted","Data":"788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d"}
Oct 31 00:15:09.969285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.969251    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerStarted","Data":"3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77"}
Oct 31 00:15:09.971383 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.971351    4466 generic.go:356] "Generic (PLEG): container finished" podID="b4d40534-5cef-4780-9d12-39785abe6b05" containerID="d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506" exitCode=60
Oct 31 00:15:09.971534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.971395    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" event={"ID":"b4d40534-5cef-4780-9d12-39785abe6b05","Type":"ContainerDied","Data":"d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506"}
Oct 31 00:15:09.971534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.971465    4466 scope.go:122] "RemoveContainer" containerID="9fa9be89037fd062c63c5b4c6347e238f7952a05c38087ae53b1e3fb21bba31d"
Oct 31 00:15:09.971872 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.971794    4466 scope.go:122] "RemoveContainer" containerID="d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506"
Oct 31 00:15:09.972222 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:09.972192    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf_odh-ai-gateway-infra(b4d40534-5cef-4780-9d12-39785abe6b05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" podUID="b4d40534-5cef-4780-9d12-39785abe6b05"
Oct 31 00:15:09.975784 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.975750    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerStarted","Data":"6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07"}
Oct 31 00:15:09.978188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.978148    4466 generic.go:356] "Generic (PLEG): container finished" podID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerID="94c9f0e8d8230ebcc54c34b7f23f4c1e2b307f9b54fac318bfb002566b1b6af4" exitCode=60
Oct 31 00:15:09.978349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.978239    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerDied","Data":"94c9f0e8d8230ebcc54c34b7f23f4c1e2b307f9b54fac318bfb002566b1b6af4"}
Oct 31 00:15:09.978349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:09.978311    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerStarted","Data":"e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310"}
Oct 31 00:15:10.011617 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:10.011577    4466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d40534_5cef_4780_9d12_39785abe6b05.slice/crio-conmon-d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:15:10.077579 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.077507    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" podStartSLOduration=3.581032037 podStartE2EDuration="10.077485372s" totalImagesPullingTime="6.496453335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:10.07587364 +0000 UTC m=+2015.366421957" watchObservedRunningTime="2026-10-31 00:15:10.077485372 +0000 UTC m=+2015.368033692"
Oct 31 00:15:10.188938 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.188836    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" podStartSLOduration=3.697773232 podStartE2EDuration="10.188820778s" totalImagesPullingTime="6.491047546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:10.185451772 +0000 UTC m=+2015.476000113" watchObservedRunningTime="2026-10-31 00:15:10.188820778 +0000 UTC m=+2015.479369095"
Oct 31 00:15:10.987737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.987696    4466 generic.go:356] "Generic (PLEG): container finished" podID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerID="09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c" exitCode=0
Oct 31 00:15:10.988227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.987750    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" event={"ID":"38d9550a-a812-4abc-84ff-f4a62648aaf6","Type":"ContainerDied","Data":"09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"}
Oct 31 00:15:10.988289 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.988229    4466 scope.go:122] "RemoveContainer" containerID="09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"
Oct 31 00:15:10.989707 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.989686    4466 generic.go:356] "Generic (PLEG): container finished" podID="736810f3-f4c6-4671-be23-dde23a413f83" containerID="788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d" exitCode=60
Oct 31 00:15:10.989820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.989765    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" event={"ID":"736810f3-f4c6-4671-be23-dde23a413f83","Type":"ContainerDied","Data":"788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d"}
Oct 31 00:15:10.989886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.989821    4466 scope.go:122] "RemoveContainer" containerID="80ee78a9ac368ea5c9d8da8e0bf0c879812a2641346ed8c5964dc314eb0bba6c"
Oct 31 00:15:10.990047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.990030    4466 scope.go:122] "RemoveContainer" containerID="788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d"
Oct 31 00:15:10.990249 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.990211    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:15:10.990343 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.990328    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:15:10.990402 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.990381    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:15:10.990528 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:10.990379    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42_odh-ai-gateway-infra(736810f3-f4c6-4671-be23-dde23a413f83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" podUID="736810f3-f4c6-4671-be23-dde23a413f83"
Oct 31 00:15:10.991792 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.991771    4466 generic.go:356] "Generic (PLEG): container finished" podID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerID="3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77" exitCode=60
Oct 31 00:15:10.991878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.991842    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerDied","Data":"3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77"}
Oct 31 00:15:10.992050 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.992036    4466 scope.go:122] "RemoveContainer" containerID="3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77"
Oct 31 00:15:10.992288 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:10.992271    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r_odh-ai-gateway-infra(a954aebe-c971-4c71-b843-ba8e8de24bc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6"
Oct 31 00:15:10.993712 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.993691    4466 scope.go:122] "RemoveContainer" containerID="d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506"
Oct 31 00:15:10.993940 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:10.993917    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf_odh-ai-gateway-infra(b4d40534-5cef-4780-9d12-39785abe6b05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" podUID="b4d40534-5cef-4780-9d12-39785abe6b05"
Oct 31 00:15:10.995487 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.995452    4466 generic.go:356] "Generic (PLEG): container finished" podID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerID="6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07" exitCode=60
Oct 31 00:15:10.995576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.995539    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerDied","Data":"6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07"}
Oct 31 00:15:10.995776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.995761    4466 scope.go:122] "RemoveContainer" containerID="6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07"
Oct 31 00:15:10.996033 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:10.996012    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk_odh-ai-gateway-infra(d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb"
Oct 31 00:15:10.997567 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.997546    4466 generic.go:356] "Generic (PLEG): container finished" podID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerID="e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310" exitCode=60
Oct 31 00:15:10.997740 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.997713    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerDied","Data":"e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310"}
Oct 31 00:15:10.997850 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:10.997835    4466 scope.go:122] "RemoveContainer" containerID="e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310"
Oct 31 00:15:10.998060 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:10.998038    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq_odh-ai-gateway-infra(e56611d6-ed45-40fa-ace0-6f1616c1f0ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca"
Oct 31 00:15:11.004352 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:11.004329    4466 scope.go:122] "RemoveContainer" containerID="1f5d3de1d2f92697dcb00826714033502d04897dba69df9ab1ce23f973c50008"
Oct 31 00:15:11.027700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:11.027673    4466 scope.go:122] "RemoveContainer" containerID="6939a12f6000ac6fb9939e9d9e47d3d453766fceda6352e57dcd462c6bba64ee"
Oct 31 00:15:11.052053 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:11.052024    4466 scope.go:122] "RemoveContainer" containerID="94c9f0e8d8230ebcc54c34b7f23f4c1e2b307f9b54fac318bfb002566b1b6af4"
Oct 31 00:15:12.006882 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.006833    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" event={"ID":"38d9550a-a812-4abc-84ff-f4a62648aaf6","Type":"ContainerStarted","Data":"f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c"}
Oct 31 00:15:12.008921 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.008898    4466 scope.go:122] "RemoveContainer" containerID="788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d"
Oct 31 00:15:12.009224 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:12.009201    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42_odh-ai-gateway-infra(736810f3-f4c6-4671-be23-dde23a413f83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" podUID="736810f3-f4c6-4671-be23-dde23a413f83"
Oct 31 00:15:12.010361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.010340    4466 scope.go:122] "RemoveContainer" containerID="3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77"
Oct 31 00:15:12.010675 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:12.010623    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r_odh-ai-gateway-infra(a954aebe-c971-4c71-b843-ba8e8de24bc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6"
Oct 31 00:15:12.012079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.012062    4466 scope.go:122] "RemoveContainer" containerID="6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07"
Oct 31 00:15:12.012309 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:12.012287    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk_odh-ai-gateway-infra(d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb"
Oct 31 00:15:12.013568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.013551    4466 scope.go:122] "RemoveContainer" containerID="e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310"
Oct 31 00:15:12.013782 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:12.013764    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq_odh-ai-gateway-infra(e56611d6-ed45-40fa-ace0-6f1616c1f0ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca"
Oct 31 00:15:12.890624 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.890587    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:15:12.892115 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.892088    4466 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:15:12.892281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:12.892136    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 00:15:13.677685 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:13.677147    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:15:13.890622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:13.890588    4466 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:15:13.890832 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:13.890682    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 00:15:14.890883 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:14.890842    4466 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:15:14.891389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:14.890908    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 00:15:15.890793 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:15.890758    4466 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:15:15.891043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:15.891022    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 00:15:16.890320 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:16.890283    4466 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:15:16.890530 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:16.890344    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 00:15:17.895887 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:17.895854    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:15:17.896913 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:17.896893    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:15:23.821358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:23.821326    4466 scope.go:122] "RemoveContainer" containerID="e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310"
Oct 31 00:15:23.821893 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:23.821425    4466 scope.go:122] "RemoveContainer" containerID="6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07"
Oct 31 00:15:24.825130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:24.825093    4466 scope.go:122] "RemoveContainer" containerID="d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506"
Oct 31 00:15:24.825557 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:24.825244    4466 scope.go:122] "RemoveContainer" containerID="788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d"
Oct 31 00:15:24.825557 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:24.825331    4466 scope.go:122] "RemoveContainer" containerID="3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77"
Oct 31 00:15:25.128796 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.128765    4466 generic.go:356] "Generic (PLEG): container finished" podID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerID="55751f3880bb3f1b19eb89e6c4f77b6fbc4e4a18197f0fe0dca35d2898eaa0bc" exitCode=60
Oct 31 00:15:25.128977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.128836    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerDied","Data":"55751f3880bb3f1b19eb89e6c4f77b6fbc4e4a18197f0fe0dca35d2898eaa0bc"}
Oct 31 00:15:25.128977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.128908    4466 scope.go:122] "RemoveContainer" containerID="6c570de451a81b581ee00c1f85d9c6b704b7f30796ee2273584c8ff79f06dd07"
Oct 31 00:15:25.129370 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.129348    4466 scope.go:122] "RemoveContainer" containerID="55751f3880bb3f1b19eb89e6c4f77b6fbc4e4a18197f0fe0dca35d2898eaa0bc"
Oct 31 00:15:25.129826 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:25.129803    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk_odh-ai-gateway-infra(d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb"
Oct 31 00:15:25.131129 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.131107    4466 generic.go:356] "Generic (PLEG): container finished" podID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerID="d7e2f335ca7afc2460245093fcc9b2b173833387e885a1a47ba931d5e89c7a38" exitCode=60
Oct 31 00:15:25.131224 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.131184    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerDied","Data":"d7e2f335ca7afc2460245093fcc9b2b173833387e885a1a47ba931d5e89c7a38"}
Oct 31 00:15:25.131571 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.131556    4466 scope.go:122] "RemoveContainer" containerID="d7e2f335ca7afc2460245093fcc9b2b173833387e885a1a47ba931d5e89c7a38"
Oct 31 00:15:25.131882 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:25.131861    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq_odh-ai-gateway-infra(e56611d6-ed45-40fa-ace0-6f1616c1f0ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca"
Oct 31 00:15:25.145837 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:25.145813    4466 scope.go:122] "RemoveContainer" containerID="e080e470bb843a2ae04a87f7df3fdffc25b81e5938f4455eda0dd061c463f310"
Oct 31 00:15:26.151140 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.151105    4466 generic.go:356] "Generic (PLEG): container finished" podID="736810f3-f4c6-4671-be23-dde23a413f83" containerID="d8250c8fc0b5cbcbafb46111ffe1d86bd10502d7a0e5b62074f17de01355f2ab" exitCode=60
Oct 31 00:15:26.151654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.151198    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" event={"ID":"736810f3-f4c6-4671-be23-dde23a413f83","Type":"ContainerDied","Data":"d8250c8fc0b5cbcbafb46111ffe1d86bd10502d7a0e5b62074f17de01355f2ab"}
Oct 31 00:15:26.151654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.151253    4466 scope.go:122] "RemoveContainer" containerID="788ed89a356955657f748070068862093b13ba5fc8cfacc59b71edd16c85d05d"
Oct 31 00:15:26.151850 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.151827    4466 scope.go:122] "RemoveContainer" containerID="d8250c8fc0b5cbcbafb46111ffe1d86bd10502d7a0e5b62074f17de01355f2ab"
Oct 31 00:15:26.156653 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:26.152386    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42_odh-ai-gateway-infra(736810f3-f4c6-4671-be23-dde23a413f83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" podUID="736810f3-f4c6-4671-be23-dde23a413f83"
Oct 31 00:15:26.168321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.167484    4466 generic.go:356] "Generic (PLEG): container finished" podID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerID="1560581b4a0c638caff8e79fe0335a98d21d20d30ce89689473b2dc49af6bd2b" exitCode=60
Oct 31 00:15:26.168452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.167563    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerDied","Data":"1560581b4a0c638caff8e79fe0335a98d21d20d30ce89689473b2dc49af6bd2b"}
Oct 31 00:15:26.169028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.169007    4466 scope.go:122] "RemoveContainer" containerID="1560581b4a0c638caff8e79fe0335a98d21d20d30ce89689473b2dc49af6bd2b"
Oct 31 00:15:26.170034 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:26.169984    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r_odh-ai-gateway-infra(a954aebe-c971-4c71-b843-ba8e8de24bc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6"
Oct 31 00:15:26.173582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.173555    4466 generic.go:356] "Generic (PLEG): container finished" podID="b4d40534-5cef-4780-9d12-39785abe6b05" containerID="c7743d9af36365adeffb7ff43e0a6f6d8dc501a3f47e746950ff69a2621d475e" exitCode=60
Oct 31 00:15:26.174284 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.173708    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" event={"ID":"b4d40534-5cef-4780-9d12-39785abe6b05","Type":"ContainerDied","Data":"c7743d9af36365adeffb7ff43e0a6f6d8dc501a3f47e746950ff69a2621d475e"}
Oct 31 00:15:26.174918 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.174901    4466 scope.go:122] "RemoveContainer" containerID="c7743d9af36365adeffb7ff43e0a6f6d8dc501a3f47e746950ff69a2621d475e"
Oct 31 00:15:26.176972 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:26.176946    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf_odh-ai-gateway-infra(b4d40534-5cef-4780-9d12-39785abe6b05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" podUID="b4d40534-5cef-4780-9d12-39785abe6b05"
Oct 31 00:15:26.184115 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.184091    4466 scope.go:122] "RemoveContainer" containerID="3979c90a4e4e86a51e8ef56dabe92d5e4c812c3ed6e1a6745010e2e159dcce77"
Oct 31 00:15:26.197383 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.197360    4466 scope.go:122] "RemoveContainer" containerID="d4016de2aacd95951d771f1a373aa4e64da65bfd5277f732ff76c4b60a706506"
Oct 31 00:15:26.267753 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.267664    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"]
Oct 31 00:15:26.378297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.378237    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"]
Oct 31 00:15:26.481425 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.481391    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:26.527502 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.527406    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k\" (UniqueName: \"kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k\") pod \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") "
Oct 31 00:15:26.527697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.527491    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca\") pod \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\" (UID: \"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb\") "
Oct 31 00:15:26.527929 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.527904    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca" pod "d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" (UID: "d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:26.528032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.528018    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:26.530341 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.530312    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k" pod "d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" (UID: "d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb"). InnerVolumeSpecName "kube-api-access-hg24k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:26.555432 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.555409    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:26.628988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.628955    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf\" (UniqueName: \"kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf\") pod \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") "
Oct 31 00:15:26.629136 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.629100    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca\") pod \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\" (UID: \"e56611d6-ed45-40fa-ace0-6f1616c1f0ca\") "
Oct 31 00:15:26.629544 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.629511    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca" pod "e56611d6-ed45-40fa-ace0-6f1616c1f0ca" (UID: "e56611d6-ed45-40fa-ace0-6f1616c1f0ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:26.629733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.629714    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg24k\" (UniqueName: \"kubernetes.io/projected/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb-kube-api-access-hg24k\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:26.629794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.629735    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:26.631274 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.631251    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf" pod "e56611d6-ed45-40fa-ace0-6f1616c1f0ca" (UID: "e56611d6-ed45-40fa-ace0-6f1616c1f0ca"). InnerVolumeSpecName "kube-api-access-db6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:26.730867 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:26.730829    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db6tf\" (UniqueName: \"kubernetes.io/projected/e56611d6-ed45-40fa-ace0-6f1616c1f0ca-kube-api-access-db6tf\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.193343 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.193258    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"
Oct 31 00:15:27.193343 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.193254    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk" event={"ID":"d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb","Type":"ContainerDied","Data":"e8a3c9b77191a2826c711497923b57a14b521fcb636528aeb48b666789e2d7a4"}
Oct 31 00:15:27.193923 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.193389    4466 scope.go:122] "RemoveContainer" containerID="55751f3880bb3f1b19eb89e6c4f77b6fbc4e4a18197f0fe0dca35d2898eaa0bc"
Oct 31 00:15:27.194782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.194559    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"
Oct 31 00:15:27.194782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.194611    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq" event={"ID":"e56611d6-ed45-40fa-ace0-6f1616c1f0ca","Type":"ContainerDied","Data":"8948dccfc3ac9f959d467abd27932df2a8a65dc8200bc15a2d0cd5645e5036a9"}
Oct 31 00:15:27.203453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.203435    4466 scope.go:122] "RemoveContainer" containerID="d7e2f335ca7afc2460245093fcc9b2b173833387e885a1a47ba931d5e89c7a38"
Oct 31 00:15:27.277516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.277472    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"]
Oct 31 00:15:27.357904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.357868    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"]
Oct 31 00:15:27.361539 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.361510    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-957019-29890095-ntqdk"]
Oct 31 00:15:27.392603 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.387745    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"]
Oct 31 00:15:27.455696 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.454243    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:27.475245 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.475208    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"]
Oct 31 00:15:27.483682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.483627    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-39e3a8-29890095-d9lwq"]
Oct 31 00:15:27.502609 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.502574    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"]
Oct 31 00:15:27.541722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.541698    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca\") pod \"736810f3-f4c6-4671-be23-dde23a413f83\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") "
Oct 31 00:15:27.542000 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.541983    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2\" (UniqueName: \"kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2\") pod \"736810f3-f4c6-4671-be23-dde23a413f83\" (UID: \"736810f3-f4c6-4671-be23-dde23a413f83\") "
Oct 31 00:15:27.542080 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.542016    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca" pod "736810f3-f4c6-4671-be23-dde23a413f83" (UID: "736810f3-f4c6-4671-be23-dde23a413f83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:27.542517 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.542496    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736810f3-f4c6-4671-be23-dde23a413f83-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.544051 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.544031    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2" pod "736810f3-f4c6-4671-be23-dde23a413f83" (UID: "736810f3-f4c6-4671-be23-dde23a413f83"). InnerVolumeSpecName "kube-api-access-82hf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:27.575172 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.575143    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:27.643073 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.643042    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7\" (UniqueName: \"kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7\") pod \"b4d40534-5cef-4780-9d12-39785abe6b05\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") "
Oct 31 00:15:27.643288 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.643270    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca\") pod \"b4d40534-5cef-4780-9d12-39785abe6b05\" (UID: \"b4d40534-5cef-4780-9d12-39785abe6b05\") "
Oct 31 00:15:27.643684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.643658    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca" pod "b4d40534-5cef-4780-9d12-39785abe6b05" (UID: "b4d40534-5cef-4780-9d12-39785abe6b05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:27.643862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.643847    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hf2\" (UniqueName: \"kubernetes.io/projected/736810f3-f4c6-4671-be23-dde23a413f83-kube-api-access-82hf2\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.643900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.643868    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d40534-5cef-4780-9d12-39785abe6b05-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.645284 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.645256    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7" pod "b4d40534-5cef-4780-9d12-39785abe6b05" (UID: "b4d40534-5cef-4780-9d12-39785abe6b05"). InnerVolumeSpecName "kube-api-access-5spl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:27.660964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.660939    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:27.744627 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.744549    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9\" (UniqueName: \"kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9\") pod \"a954aebe-c971-4c71-b843-ba8e8de24bc6\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") "
Oct 31 00:15:27.744794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.744674    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca\") pod \"a954aebe-c971-4c71-b843-ba8e8de24bc6\" (UID: \"a954aebe-c971-4c71-b843-ba8e8de24bc6\") "
Oct 31 00:15:27.745052 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.745007    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca" pod "a954aebe-c971-4c71-b843-ba8e8de24bc6" (UID: "a954aebe-c971-4c71-b843-ba8e8de24bc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:27.745238 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.745221    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a954aebe-c971-4c71-b843-ba8e8de24bc6-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.745279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.745241    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5spl7\" (UniqueName: \"kubernetes.io/projected/b4d40534-5cef-4780-9d12-39785abe6b05-kube-api-access-5spl7\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:27.746765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.746744    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9" pod "a954aebe-c971-4c71-b843-ba8e8de24bc6" (UID: "a954aebe-c971-4c71-b843-ba8e8de24bc6"). InnerVolumeSpecName "kube-api-access-wbth9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:27.847013 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:27.846969    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbth9\" (UniqueName: \"kubernetes.io/projected/a954aebe-c971-4c71-b843-ba8e8de24bc6-kube-api-access-wbth9\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:28.202735 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.202691    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf" event={"ID":"b4d40534-5cef-4780-9d12-39785abe6b05","Type":"ContainerDied","Data":"ab3431790bfa8f261fd7f4c10aee8ca668148c8615149789c51d7b1e8303917e"}
Oct 31 00:15:28.202735 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.202724    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"
Oct 31 00:15:28.203263 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.202772    4466 scope.go:122] "RemoveContainer" containerID="c7743d9af36365adeffb7ff43e0a6f6d8dc501a3f47e746950ff69a2621d475e"
Oct 31 00:15:28.207261 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.207226    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42" event={"ID":"736810f3-f4c6-4671-be23-dde23a413f83","Type":"ContainerDied","Data":"5b815e31215269173499945739a9868c02a195c91bdb6b41d359674925d085d9"}
Oct 31 00:15:28.207388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.207237    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"
Oct 31 00:15:28.208806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.208786    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r" event={"ID":"a954aebe-c971-4c71-b843-ba8e8de24bc6","Type":"ContainerDied","Data":"dbfb51d1b13fd94b9817d574a227284f2eaa35b19b897455ab9d1f8a4b42869a"}
Oct 31 00:15:28.208907 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.208805    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"
Oct 31 00:15:28.215006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.214983    4466 scope.go:122] "RemoveContainer" containerID="d8250c8fc0b5cbcbafb46111ffe1d86bd10502d7a0e5b62074f17de01355f2ab"
Oct 31 00:15:28.225105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.225083    4466 scope.go:122] "RemoveContainer" containerID="1560581b4a0c638caff8e79fe0335a98d21d20d30ce89689473b2dc49af6bd2b"
Oct 31 00:15:28.360302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.360178    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"]
Oct 31 00:15:28.364063 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.364034    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fe01fa-29890095-xb75r"]
Oct 31 00:15:28.468985 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.468860    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"]
Oct 31 00:15:28.472946 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.472911    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d9e782-29890095-jxj42"]
Oct 31 00:15:28.578114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.578082    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"]
Oct 31 00:15:28.581460 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.581429    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46b586-29890095-jkdqf"]
Oct 31 00:15:28.828871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.828786    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736810f3-f4c6-4671-be23-dde23a413f83" path="/var/lib/kubelet/pods/736810f3-f4c6-4671-be23-dde23a413f83/volumes"
Oct 31 00:15:28.829304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.829290    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" path="/var/lib/kubelet/pods/a954aebe-c971-4c71-b843-ba8e8de24bc6/volumes"
Oct 31 00:15:28.829697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.829680    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" path="/var/lib/kubelet/pods/b4d40534-5cef-4780-9d12-39785abe6b05/volumes"
Oct 31 00:15:28.830410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.830396    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" path="/var/lib/kubelet/pods/d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb/volumes"
Oct 31 00:15:28.830832 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:28.830819    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" path="/var/lib/kubelet/pods/e56611d6-ed45-40fa-ace0-6f1616c1f0ca/volumes"
Oct 31 00:15:38.318721 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:38.318684    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:15:40.058082 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058043    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"]
Oct 31 00:15:40.058785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058767    4466 cpu_manager.go:414] "containerMap: removing container" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:15:40.058785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058785    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058799    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058804    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058813    4466 cpu_manager.go:414] "containerMap: removing container" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058819    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058827    4466 cpu_manager.go:414] "containerMap: removing container" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058832    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058855    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058859    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058866    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058871    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:15:40.058880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058882    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058887    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058900    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058904    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058913    4466 cpu_manager.go:414] "containerMap: removing container" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058917    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058926    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058931    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058940    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.058944    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059024    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059033    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059038    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059045    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059050    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059055    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059064    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059072    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059079    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d40534-5cef-4780-9d12-39785abe6b05" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059084    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:15:40.059204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.059090    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:15:40.064529 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.064509    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.141028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.140977    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74sh\" (UniqueName: \"kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.141240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.141183    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.141378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.141358    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.141582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.141557    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.160708 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.160677    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-460d4e-serving-cert\""
Oct 31 00:15:40.164611 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.164585    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"]
Oct 31 00:15:40.243094 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.243053    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.243291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.243134    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74sh\" (UniqueName: \"kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.243291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.243262    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.243439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.243422    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.243571 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:40.243551    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-serving-cert: secret "maas-api-e2e-inf-b-460d4e-serving-cert" not found
Oct 31 00:15:40.243675 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:15:40.243663    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls podName:587216ab-cc26-46fa-b941-c441f1ec827e nodeName:}" failed. No retries permitted until 2026-10-31 00:15:40.74360792 +0000 UTC m=+2046.034156234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls") pod "maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" (UID: "587216ab-cc26-46fa-b941-c441f1ec827e") : secret "maas-api-e2e-inf-b-460d4e-serving-cert" not found
Oct 31 00:15:40.243919 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.243902    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.245532 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.245507    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.302039 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.302009    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74sh\" (UniqueName: \"kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.756833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.756799    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.759617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.759574    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\") pod \"maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:40.978551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:40.978511    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:41.362886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:41.362854    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"]
Oct 31 00:15:41.363617 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:15:41.363593    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587216ab_cc26_46fa_b941_c441f1ec827e.slice/crio-3da6bacf0dd01e17b30194e3c0410f52f4422098689f4e473357f6291b66e9fb WatchSource:0}: Error finding container 3da6bacf0dd01e17b30194e3c0410f52f4422098689f4e473357f6291b66e9fb: Status 404 returned error can't find the container with id 3da6bacf0dd01e17b30194e3c0410f52f4422098689f4e473357f6291b66e9fb
Oct 31 00:15:42.310455 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:42.310417    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" event={"ID":"587216ab-cc26-46fa-b941-c441f1ec827e","Type":"ContainerStarted","Data":"3da6bacf0dd01e17b30194e3c0410f52f4422098689f4e473357f6291b66e9fb"}
Oct 31 00:15:43.318082 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:43.318043    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" event={"ID":"587216ab-cc26-46fa-b941-c441f1ec827e","Type":"ContainerStarted","Data":"5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd"}
Oct 31 00:15:43.318580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:43.318228    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:15:43.433163 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:43.433111    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" podStartSLOduration=2.407583062 podStartE2EDuration="3.433094051s" totalImagesPullingTime="1.025510989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:43.429762316 +0000 UTC m=+2048.720310637" watchObservedRunningTime="2026-10-31 00:15:43.433094051 +0000 UTC m=+2048.723642370"
Oct 31 00:15:49.331248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:15:49.331215    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:16:03.214690 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.214651    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:03.215131 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.214718    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:03.457870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.457831    4466 generic.go:356] "Generic (PLEG): container finished" podID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerID="9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3" exitCode=0
Oct 31 00:16:03.458047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.457906    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" event={"ID":"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4","Type":"ContainerDied","Data":"9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"}
Oct 31 00:16:03.458391 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.458373    4466 scope.go:122] "RemoveContainer" containerID="9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"
Oct 31 00:16:03.460191 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.460165    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:03.460261 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.460232    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:03.460297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:03.460264    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:04.471251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:04.471213    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" event={"ID":"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4","Type":"ContainerStarted","Data":"21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9"}
Oct 31 00:16:04.494689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:04.494654    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:16:04.494988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:04.494966    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:04.495040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:04.495016    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:05.495188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:05.495154    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:05.495603 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:05.495208    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:06.495062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:06.495026    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:06.495232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:06.495085    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:07.495463 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:07.495431    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:07.495893 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:07.495484    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:08.321038 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:16:08.321000    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:16:08.496486 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:08.496198    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:08.496486 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:08.496263    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:09.495056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:09.495016    4466 patch_prober.go:29] interesting pod/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 00:16:09.495252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:09.495078    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 00:16:10.499943 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:10.499901    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:16:10.500334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:10.500299    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:16:10.501065 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:10.501046    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:16:20.996730 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:20.996692    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"]
Oct 31 00:16:21.097686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:21.097646    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"]
Oct 31 00:16:25.997332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:25.997286    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" podUID="5e6f4213-ea07-410c-917d-127636230349" containerName="payload-processing" containerID="cri-o://d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d" gracePeriod=25
Oct 31 00:16:26.098176 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.098124    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" podUID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" containerName="payload-pre-processing" containerID="cri-o://41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea" gracePeriod=25
Oct 31 00:16:26.181224 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:16:26.181182    4466 logging.go:55] [core] [Channel #438 SubChannel #439]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.79:50052", ServerName: "10.133.0.79:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.79:50052: connect: connection refused"
Oct 31 00:16:26.272221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.272191    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:16:26.398685 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.398338    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume\") pod \"5e6f4213-ea07-410c-917d-127636230349\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") "
Oct 31 00:16:26.398685 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.398538    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs\" (UniqueName: \"kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs\") pod \"5e6f4213-ea07-410c-917d-127636230349\" (UID: \"5e6f4213-ea07-410c-917d-127636230349\") "
Oct 31 00:16:26.400207 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.400174    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume" pod "5e6f4213-ea07-410c-917d-127636230349" (UID: "5e6f4213-ea07-410c-917d-127636230349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:26.400449 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.400422    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6f4213-ea07-410c-917d-127636230349-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:26.406131 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.406101    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs" pod "5e6f4213-ea07-410c-917d-127636230349" (UID: "5e6f4213-ea07-410c-917d-127636230349"). InnerVolumeSpecName "kube-api-access-hl5qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:26.429187 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.429154    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:16:26.501833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.501743    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume\") pod \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") "
Oct 31 00:16:26.501988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.501944    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w\" (UniqueName: \"kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w\") pod \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\" (UID: \"d9a9e180-0d68-4a52-95fa-136ddc8bb425\") "
Oct 31 00:16:26.502125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.502103    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume" pod "d9a9e180-0d68-4a52-95fa-136ddc8bb425" (UID: "d9a9e180-0d68-4a52-95fa-136ddc8bb425"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:26.502442 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.502422    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a9e180-0d68-4a52-95fa-136ddc8bb425-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:26.502492 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.502441    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5qs\" (UniqueName: \"kubernetes.io/projected/5e6f4213-ea07-410c-917d-127636230349-kube-api-access-hl5qs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:26.504022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.503998    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w" pod "d9a9e180-0d68-4a52-95fa-136ddc8bb425" (UID: "d9a9e180-0d68-4a52-95fa-136ddc8bb425"). InnerVolumeSpecName "kube-api-access-mz85w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:26.604222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.604181    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz85w\" (UniqueName: \"kubernetes.io/projected/d9a9e180-0d68-4a52-95fa-136ddc8bb425-kube-api-access-mz85w\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:26.628038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.628004    4466 generic.go:356] "Generic (PLEG): container finished" podID="5e6f4213-ea07-410c-917d-127636230349" containerID="d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d" exitCode=0
Oct 31 00:16:26.628258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.628057    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"
Oct 31 00:16:26.628258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.628086    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" event={"ID":"5e6f4213-ea07-410c-917d-127636230349","Type":"ContainerDied","Data":"d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d"}
Oct 31 00:16:26.628258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.628142    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" event={"ID":"5e6f4213-ea07-410c-917d-127636230349","Type":"ContainerDied","Data":"1834294ad18bc7eba0b4b3bf5568649045ebcd7e00281bac52ef47dfefd8133d"}
Oct 31 00:16:26.628258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.628173    4466 scope.go:122] "RemoveContainer" containerID="d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d"
Oct 31 00:16:26.629623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.629600    4466 generic.go:356] "Generic (PLEG): container finished" podID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" containerID="41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea" exitCode=0
Oct 31 00:16:26.629763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.629668    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" event={"ID":"d9a9e180-0d68-4a52-95fa-136ddc8bb425","Type":"ContainerDied","Data":"41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea"}
Oct 31 00:16:26.629763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.629713    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" event={"ID":"d9a9e180-0d68-4a52-95fa-136ddc8bb425","Type":"ContainerDied","Data":"44980f62cf16e6a98da009728bc1bda46f695a5dd6d35b64a3cb502c54f297f3"}
Oct 31 00:16:26.629763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.629674    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"
Oct 31 00:16:26.638804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.638620    4466 scope.go:122] "RemoveContainer" containerID="d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d"
Oct 31 00:16:26.639185 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:16:26.639152    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d\": container with ID starting with d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d not found: ID does not exist" containerID="d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d"
Oct 31 00:16:26.639246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.639190    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d"} err="failed to get container status \"d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d\": rpc error: code = NotFound desc = could not find container \"d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d\": container with ID starting with d0e458415448bbcd2b5a778136ce3c58054b82c3d9d4cb10d7c70401a4b0057d not found: ID does not exist"
Oct 31 00:16:26.639246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.639217    4466 scope.go:122] "RemoveContainer" containerID="41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea"
Oct 31 00:16:26.649260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.649237    4466 scope.go:122] "RemoveContainer" containerID="41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea"
Oct 31 00:16:26.649594 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:16:26.649569    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea\": container with ID starting with 41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea not found: ID does not exist" containerID="41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea"
Oct 31 00:16:26.649661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.649604    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea"} err="failed to get container status \"41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea\": rpc error: code = NotFound desc = could not find container \"41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea\": container with ID starting with 41cb7997eadf854de707ec3d588627c75ccdbfada332a5d40be397c17762c1ea not found: ID does not exist"
Oct 31 00:16:26.786215 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.786115    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"]
Oct 31 00:16:26.792766 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.792737    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn"]
Oct 31 00:16:26.833133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.833097    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6f4213-ea07-410c-917d-127636230349" path="/var/lib/kubelet/pods/5e6f4213-ea07-410c-917d-127636230349/volumes"
Oct 31 00:16:26.894866 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.894830    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"]
Oct 31 00:16:26.901355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:26.901317    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs"]
Oct 31 00:16:27.181092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:27.181048    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a3dba-57c6cf5fd4-n7zpn" podUID="5e6f4213-ea07-410c-917d-127636230349" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.80:50052\" within 1s: context deadline exceeded"
Oct 31 00:16:27.181497 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:16:27.181137    4466 logging.go:55] [core] [Channel #437 SubChannel #440]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.80:50052", ServerName: "10.133.0.80:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.80:50052: operation was canceled"
Oct 31 00:16:27.181497 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:27.181048    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a3dba-576dcfd7cf-6x2bs" podUID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.79:50052\" within 1s: context deadline exceeded"
Oct 31 00:16:28.829681 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:28.829575    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" path="/var/lib/kubelet/pods/d9a9e180-0d68-4a52-95fa-136ddc8bb425/volumes"
Oct 31 00:16:34.474983 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.474946    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"]
Oct 31 00:16:34.475541 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475527    4466 cpu_manager.go:414] "containerMap: removing container" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:16:34.475587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475542    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:16:34.475587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475557    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:16:34.475587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475562    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:16:34.475587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475573    4466 cpu_manager.go:414] "containerMap: removing container" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:16:34.475587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475578    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475600    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475605    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475618    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" containerName="payload-pre-processing"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475623    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" containerName="payload-pre-processing"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475657    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5e6f4213-ea07-410c-917d-127636230349" containerName="payload-processing"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475664    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6f4213-ea07-410c-917d-127636230349" containerName="payload-processing"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475765    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="736810f3-f4c6-4671-be23-dde23a413f83" containerName="cleanup"
Oct 31 00:16:34.475780 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475778    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56611d6-ed45-40fa-ace0-6f1616c1f0ca" containerName="cleanup"
Oct 31 00:16:34.476022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475787    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a9e180-0d68-4a52-95fa-136ddc8bb425" containerName="payload-pre-processing"
Oct 31 00:16:34.476022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475795    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a954aebe-c971-4c71-b843-ba8e8de24bc6" containerName="cleanup"
Oct 31 00:16:34.476022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475802    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bce1b4-77cd-4f07-b8c7-28b0a2ac1bcb" containerName="cleanup"
Oct 31 00:16:34.476022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.475808    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6f4213-ea07-410c-917d-127636230349" containerName="payload-processing"
Oct 31 00:16:34.479409 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.479392    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.579649 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.579603    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"]
Oct 31 00:16:34.584276 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.584238    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"]
Oct 31 00:16:34.584450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.584365    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.618163 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618125    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618192    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618263    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618335    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618438    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618464    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618487    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618537    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.618782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.618586    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wpd\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.632719 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.632681    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"]
Oct 31 00:16:34.719545 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719511    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.719762 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719551    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.719762 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719600    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.719762 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719661    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.719953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719819    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.719953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719886    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.719953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719930    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.719967    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720011    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720012    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720038    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wpd\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720093 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720075    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720108    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720173    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720308    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720326    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720411    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720451    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkd6g\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720514    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720586    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720647    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.720886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.720702    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.721209 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.721064    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.722879 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.722854    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.723020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.722995    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.780863 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.780781    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.781004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.780873    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wpd\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.793452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.793419    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:34.822360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822326    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822542 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822461    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkd6g\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822542 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822512    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822699 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822545    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822699 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822583    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822711    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822860 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822806    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822921 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822899    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.822980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.822931    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.823431 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.823400    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.823669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.823628    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.823755 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.823691    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.823904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.823880    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.823974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.823923    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.827360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.827333    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.828510 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.828488    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.883155 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.883117    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.884425 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.884397    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkd6g\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:34.910017 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:34.909986    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:35.224335 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.224305    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"]
Oct 31 00:16:35.225105 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:16:35.225076    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfa9d58_8c7b_4eff_85b5_52de57f3d9c7.slice/crio-36d282285c28629d7776e71862bd2fb74556d2994947413f0fac4d1bc5532245 WatchSource:0}: Error finding container 36d282285c28629d7776e71862bd2fb74556d2994947413f0fac4d1bc5532245: Status 404 returned error can't find the container with id 36d282285c28629d7776e71862bd2fb74556d2994947413f0fac4d1bc5532245
Oct 31 00:16:35.228053 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.228018    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:35.228138 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.228090    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:35.228138 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.228120    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:35.321399 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:16:35.321372    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73d0d2b_337d_40a3_a693_35468d86ab39.slice/crio-f87d4f657e2e88f0635b628415a5caae47f3e71751152a1ced1f3bcc01702368 WatchSource:0}: Error finding container f87d4f657e2e88f0635b628415a5caae47f3e71751152a1ced1f3bcc01702368: Status 404 returned error can't find the container with id f87d4f657e2e88f0635b628415a5caae47f3e71751152a1ced1f3bcc01702368
Oct 31 00:16:35.322900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.322870    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"]
Oct 31 00:16:35.324281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.324249    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:35.324361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.324314    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:35.324361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.324343    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:16:35.716770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.711002    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" event={"ID":"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7","Type":"ContainerStarted","Data":"8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77"}
Oct 31 00:16:35.716770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.711091    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" event={"ID":"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7","Type":"ContainerStarted","Data":"36d282285c28629d7776e71862bd2fb74556d2994947413f0fac4d1bc5532245"}
Oct 31 00:16:35.716770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.713153    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" event={"ID":"e73d0d2b-337d-40a3-a693-35468d86ab39","Type":"ContainerStarted","Data":"03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e"}
Oct 31 00:16:35.716770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.713198    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" event={"ID":"e73d0d2b-337d-40a3-a693-35468d86ab39","Type":"ContainerStarted","Data":"f87d4f657e2e88f0635b628415a5caae47f3e71751152a1ced1f3bcc01702368"}
Oct 31 00:16:35.794430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.794394    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:35.827730 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.827674    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podStartSLOduration=1.827655689 podStartE2EDuration="1.827655689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:35.824623586 +0000 UTC m=+2101.115171899" watchObservedRunningTime="2026-10-31 00:16:35.827655689 +0000 UTC m=+2101.118204010"
Oct 31 00:16:35.910568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.910537    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:35.939461 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:35.939398    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podStartSLOduration=1.9393790050000002 podStartE2EDuration="1.939379005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:35.938732514 +0000 UTC m=+2101.229280834" watchObservedRunningTime="2026-10-31 00:16:35.939379005 +0000 UTC m=+2101.229927325"
Oct 31 00:16:36.795453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:36.795418    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:16:36.795896 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:36.795468    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:16:36.912550 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:36.912513    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:16:36.912757 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:36.912575    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 00:16:37.794022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:37.793985    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:16:37.794218 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:37.794055    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:16:37.911301 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:37.911266    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:16:37.911723 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:37.911328    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 00:16:38.318939 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:16:38.318899    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:16:38.806704 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:38.806620    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:16:38.806926 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:38.806716    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:16:38.913992 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:38.913949    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:16:38.914536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:38.914010    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 00:16:39.819661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:39.807121    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:16:39.819661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:39.807198    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:16:39.910643 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:39.910597    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:16:39.910826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:39.910666    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 00:16:40.797629 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:40.797573    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:16:40.798164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:40.797719    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:16:40.910410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:40.910377    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:16:40.910589 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:40.910439    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 00:16:41.794490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:41.794453    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 00:16:41.794688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:41.794515    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 00:16:41.910330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:41.910295    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:16:41.910750 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:41.910365    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 00:16:42.798836 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:42.798805    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:42.799247 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:42.799226    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:42.799866 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:42.799841    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:16:42.916029 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:42.916002    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:42.916392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:42.916293    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:16:42.916952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:16:42.916931    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:17:00.046841 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:00.046807    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"]
Oct 31 00:17:05.048012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.047964    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2" podUID="fa15182f-8783-49ce-a4e2-a32de14f12e5" containerName="payload-processing" containerID="cri-o://5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1" gracePeriod=25
Oct 31 00:17:05.450593 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.450564    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:17:05.623001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.622975    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c\" (UniqueName: \"kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c\") pod \"fa15182f-8783-49ce-a4e2-a32de14f12e5\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") "
Oct 31 00:17:05.623244 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.623230    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume\") pod \"fa15182f-8783-49ce-a4e2-a32de14f12e5\" (UID: \"fa15182f-8783-49ce-a4e2-a32de14f12e5\") "
Oct 31 00:17:05.624097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.624078    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume" pod "fa15182f-8783-49ce-a4e2-a32de14f12e5" (UID: "fa15182f-8783-49ce-a4e2-a32de14f12e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:05.626359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.626336    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c" pod "fa15182f-8783-49ce-a4e2-a32de14f12e5" (UID: "fa15182f-8783-49ce-a4e2-a32de14f12e5"). InnerVolumeSpecName "kube-api-access-xhr8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:05.724881 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.724845    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhr8c\" (UniqueName: \"kubernetes.io/projected/fa15182f-8783-49ce-a4e2-a32de14f12e5-kube-api-access-xhr8c\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:05.724881 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:05.724876    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa15182f-8783-49ce-a4e2-a32de14f12e5-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:06.115079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.115045    4466 generic.go:356] "Generic (PLEG): container finished" podID="fa15182f-8783-49ce-a4e2-a32de14f12e5" containerID="5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1" exitCode=0
Oct 31 00:17:06.115561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.115103    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"
Oct 31 00:17:06.115561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.115128    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2" event={"ID":"fa15182f-8783-49ce-a4e2-a32de14f12e5","Type":"ContainerDied","Data":"5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1"}
Oct 31 00:17:06.115561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.115196    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2" event={"ID":"fa15182f-8783-49ce-a4e2-a32de14f12e5","Type":"ContainerDied","Data":"e1e7d35035a5cd6254e8b3c1f97caab91b21043732dea2f3d6d39db39ed5999c"}
Oct 31 00:17:06.115561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.115229    4466 scope.go:122] "RemoveContainer" containerID="5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1"
Oct 31 00:17:06.127142 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.127118    4466 scope.go:122] "RemoveContainer" containerID="5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1"
Oct 31 00:17:06.127456 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:06.127434    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1\": container with ID starting with 5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1 not found: ID does not exist" containerID="5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1"
Oct 31 00:17:06.127503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.127465    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1"} err="failed to get container status \"5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1\": rpc error: code = NotFound desc = could not find container \"5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1\": container with ID starting with 5608d583614c91afd9cb58d27c9f056ad159c83efb59ca61a66b07685024e2d1 not found: ID does not exist"
Oct 31 00:17:06.272551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.272515    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"]
Oct 31 00:17:06.318909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.318868    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-348a33-7f4b555888-dk2l2"]
Oct 31 00:17:06.829833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:06.829801    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa15182f-8783-49ce-a4e2-a32de14f12e5" path="/var/lib/kubelet/pods/fa15182f-8783-49ce-a4e2-a32de14f12e5/volumes"
Oct 31 00:17:08.317990 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:08.317950    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:17:12.461771 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:12.461729    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"]
Oct 31 00:17:14.526339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:14.526294    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"]
Oct 31 00:17:14.527376 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:14.526505    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy" containerID="cri-o://21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9" gracePeriod=30
Oct 31 00:17:17.470657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.468711    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v" podUID="a47abc60-9568-43e1-82c7-e937af2ce01f" containerName="payload-processing" containerID="cri-o://47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6" gracePeriod=25
Oct 31 00:17:17.936951 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.936920    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:17:17.952092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.949996    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume\") pod \"a47abc60-9568-43e1-82c7-e937af2ce01f\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") "
Oct 31 00:17:17.952092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.950497    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume" pod "a47abc60-9568-43e1-82c7-e937af2ce01f" (UID: "a47abc60-9568-43e1-82c7-e937af2ce01f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:17.952092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.950545    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw\" (UniqueName: \"kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw\") pod \"a47abc60-9568-43e1-82c7-e937af2ce01f\" (UID: \"a47abc60-9568-43e1-82c7-e937af2ce01f\") "
Oct 31 00:17:17.952092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.951713    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47abc60-9568-43e1-82c7-e937af2ce01f-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:17.954443 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:17.954420    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw" pod "a47abc60-9568-43e1-82c7-e937af2ce01f" (UID: "a47abc60-9568-43e1-82c7-e937af2ce01f"). InnerVolumeSpecName "kube-api-access-8rlbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:18.053698 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.053657    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rlbw\" (UniqueName: \"kubernetes.io/projected/a47abc60-9568-43e1-82c7-e937af2ce01f-kube-api-access-8rlbw\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:18.197996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.197913    4466 generic.go:356] "Generic (PLEG): container finished" podID="a47abc60-9568-43e1-82c7-e937af2ce01f" containerID="47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6" exitCode=0
Oct 31 00:17:18.197996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.197950    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v" event={"ID":"a47abc60-9568-43e1-82c7-e937af2ce01f","Type":"ContainerDied","Data":"47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6"}
Oct 31 00:17:18.198198 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.198011    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v" event={"ID":"a47abc60-9568-43e1-82c7-e937af2ce01f","Type":"ContainerDied","Data":"5a4b6a0a8a4c1b7296d98c57784e0c150ca5d984aaaf773c6e042580a34f117f"}
Oct 31 00:17:18.198198 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.197966    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"
Oct 31 00:17:18.198198 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.198033    4466 scope.go:122] "RemoveContainer" containerID="47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6"
Oct 31 00:17:18.209572 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.209550    4466 scope.go:122] "RemoveContainer" containerID="47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6"
Oct 31 00:17:18.209891 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:18.209870    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6\": container with ID starting with 47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6 not found: ID does not exist" containerID="47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6"
Oct 31 00:17:18.209971 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.209900    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6"} err="failed to get container status \"47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6\": rpc error: code = NotFound desc = could not find container \"47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6\": container with ID starting with 47f2632c978154a2245d26e5f13102cb05ca9e0f667b3fe39482e6c64a3d81b6 not found: ID does not exist"
Oct 31 00:17:18.354463 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.354424    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"]
Oct 31 00:17:18.358726 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.358699    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a251fdfb-6fb6bf7d6b-w2z4v"]
Oct 31 00:17:18.829338 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:18.829258    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47abc60-9568-43e1-82c7-e937af2ce01f" path="/var/lib/kubelet/pods/a47abc60-9568-43e1-82c7-e937af2ce01f/volumes"
Oct 31 00:17:19.807765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.807694    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:17:19.882549 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882517    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882614    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882741    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882788    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882811    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882835    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882862    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882905    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.882970    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert\") pod \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\" (UID: \"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4\") "
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883007    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:19.883043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883035    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:19.883529 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883182    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:19.883529 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883215    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:19.883529 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883432    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:19.883933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883904    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.884036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883937    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.884036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883952    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.884036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883966    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.884036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.883980    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.885089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.885065    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:19.885198 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.885074    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:19.885248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.885226    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "kube-api-access-t2zrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:19.885827 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.885810    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo" pod "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" (UID: "b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:17:19.985281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.985244    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2zrf\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-kube-api-access-t2zrf\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.985281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.985281    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.985527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.985294    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:19.985527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:19.985309    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:20.215270 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.215230    4466 generic.go:356] "Generic (PLEG): container finished" podID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerID="21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9" exitCode=0
Oct 31 00:17:20.215476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.215287    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"
Oct 31 00:17:20.215476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.215311    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" event={"ID":"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4","Type":"ContainerDied","Data":"21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9"}
Oct 31 00:17:20.215476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.215385    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4" event={"ID":"b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4","Type":"ContainerDied","Data":"98755c0957c73561199dd74de555494cfd82aa3357002e0938dbbcd1006e9112"}
Oct 31 00:17:20.215476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.215419    4466 scope.go:122] "RemoveContainer" containerID="21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9"
Oct 31 00:17:20.227089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.227071    4466 scope.go:122] "RemoveContainer" containerID="9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"
Oct 31 00:17:20.236716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.236695    4466 scope.go:122] "RemoveContainer" containerID="21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9"
Oct 31 00:17:20.236994 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:20.236973    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9\": container with ID starting with 21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9 not found: ID does not exist" containerID="21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9"
Oct 31 00:17:20.237041 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.237004    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9"} err="failed to get container status \"21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9\": rpc error: code = NotFound desc = could not find container \"21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9\": container with ID starting with 21150e57082d19834ca55550397f2a5d5a1851926e8473920098bec02dcfc9e9 not found: ID does not exist"
Oct 31 00:17:20.237041 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.237021    4466 scope.go:122] "RemoveContainer" containerID="9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"
Oct 31 00:17:20.237281 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:20.237265    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3\": container with ID starting with 9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3 not found: ID does not exist" containerID="9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"
Oct 31 00:17:20.237318 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.237285    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3"} err="failed to get container status \"9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3\": rpc error: code = NotFound desc = could not find container \"9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3\": container with ID starting with 9b8ea8b23e4369f5429e07d0c77341aa11799ff3f82b7b413bd8fea9618739c3 not found: ID does not exist"
Oct 31 00:17:20.406347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.406314    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"]
Oct 31 00:17:20.452818 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.452785    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-348a33-openshift-default-5775c46d7-w9cm4"]
Oct 31 00:17:20.828260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:20.828215    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" path="/var/lib/kubelet/pods/b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4/volumes"
Oct 31 00:17:36.893405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:36.893362    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"]
Oct 31 00:17:36.992858 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:36.992822    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"]
Oct 31 00:17:38.388362 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:38.388322    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:17:39.301798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:39.301764    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"]
Oct 31 00:17:41.893711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:41.893672    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss" podUID="02028cc9-2ea9-437b-9d75-b6221a81ce43" containerName="payload-processing" containerID="cri-o://2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff" gracePeriod=25
Oct 31 00:17:41.993182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:41.993112    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf" podUID="78c24278-cb5d-4762-8841-5c0fcf8127ad" containerName="payload-pre-processing" containerID="cri-o://e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f" gracePeriod=25
Oct 31 00:17:42.194886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.194856    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:17:42.288168 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.288143    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:17:42.358066 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.358029    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk\" (UniqueName: \"kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk\") pod \"02028cc9-2ea9-437b-9d75-b6221a81ce43\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") "
Oct 31 00:17:42.358266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.358169    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume\") pod \"78c24278-cb5d-4762-8841-5c0fcf8127ad\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") "
Oct 31 00:17:42.358266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.358219    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume\") pod \"02028cc9-2ea9-437b-9d75-b6221a81ce43\" (UID: \"02028cc9-2ea9-437b-9d75-b6221a81ce43\") "
Oct 31 00:17:42.358398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.358386    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b\" (UniqueName: \"kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b\") pod \"78c24278-cb5d-4762-8841-5c0fcf8127ad\" (UID: \"78c24278-cb5d-4762-8841-5c0fcf8127ad\") "
Oct 31 00:17:42.358592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.358565    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume" pod "78c24278-cb5d-4762-8841-5c0fcf8127ad" (UID: "78c24278-cb5d-4762-8841-5c0fcf8127ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:42.358722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.358575    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume" pod "02028cc9-2ea9-437b-9d75-b6221a81ce43" (UID: "02028cc9-2ea9-437b-9d75-b6221a81ce43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:42.359040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.359019    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c24278-cb5d-4762-8841-5c0fcf8127ad-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.359086 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.359048    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02028cc9-2ea9-437b-9d75-b6221a81ce43-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.360315 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.360294    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk" pod "02028cc9-2ea9-437b-9d75-b6221a81ce43" (UID: "02028cc9-2ea9-437b-9d75-b6221a81ce43"). InnerVolumeSpecName "kube-api-access-22csk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:42.360413 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.360355    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b" pod "78c24278-cb5d-4762-8841-5c0fcf8127ad" (UID: "78c24278-cb5d-4762-8841-5c0fcf8127ad"). InnerVolumeSpecName "kube-api-access-qwz5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:42.382070 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.382042    4466 generic.go:356] "Generic (PLEG): container finished" podID="02028cc9-2ea9-437b-9d75-b6221a81ce43" containerID="2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff" exitCode=0
Oct 31 00:17:42.382266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.382087    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"
Oct 31 00:17:42.382266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.382112    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss" event={"ID":"02028cc9-2ea9-437b-9d75-b6221a81ce43","Type":"ContainerDied","Data":"2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff"}
Oct 31 00:17:42.382266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.382167    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss" event={"ID":"02028cc9-2ea9-437b-9d75-b6221a81ce43","Type":"ContainerDied","Data":"bb0b4f2f118554877e6ea0527ab31257981e32b256892de98cd2ffb69412790d"}
Oct 31 00:17:42.382266 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.382198    4466 scope.go:122] "RemoveContainer" containerID="2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff"
Oct 31 00:17:42.383686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.383663    4466 generic.go:356] "Generic (PLEG): container finished" podID="78c24278-cb5d-4762-8841-5c0fcf8127ad" containerID="e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f" exitCode=0
Oct 31 00:17:42.383817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.383692    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf" event={"ID":"78c24278-cb5d-4762-8841-5c0fcf8127ad","Type":"ContainerDied","Data":"e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f"}
Oct 31 00:17:42.383817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.383715    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"
Oct 31 00:17:42.383817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.383746    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf" event={"ID":"78c24278-cb5d-4762-8841-5c0fcf8127ad","Type":"ContainerDied","Data":"d5f080276754e2b0d7921be73810c38c0fbf9b1277a3a03d4a701d48d08a2e51"}
Oct 31 00:17:42.393452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.393430    4466 scope.go:122] "RemoveContainer" containerID="2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff"
Oct 31 00:17:42.393821 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:42.393796    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff\": container with ID starting with 2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff not found: ID does not exist" containerID="2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff"
Oct 31 00:17:42.393901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.393831    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff"} err="failed to get container status \"2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff\": rpc error: code = NotFound desc = could not find container \"2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff\": container with ID starting with 2badcdf0f3a112923394ce2350db48873e4c58e5469fd6141c32f5aa24ecd5ff not found: ID does not exist"
Oct 31 00:17:42.393901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.393855    4466 scope.go:122] "RemoveContainer" containerID="e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f"
Oct 31 00:17:42.404253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.404235    4466 scope.go:122] "RemoveContainer" containerID="e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f"
Oct 31 00:17:42.404705 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:42.404618    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f\": container with ID starting with e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f not found: ID does not exist" containerID="e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f"
Oct 31 00:17:42.404769 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.404717    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f"} err="failed to get container status \"e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f\": rpc error: code = NotFound desc = could not find container \"e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f\": container with ID starting with e0ed0549777b312d6c7867a2f86ab70cacd30430473ce1320305a89f3176889f not found: ID does not exist"
Oct 31 00:17:42.460340 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.460249    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwz5b\" (UniqueName: \"kubernetes.io/projected/78c24278-cb5d-4762-8841-5c0fcf8127ad-kube-api-access-qwz5b\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.460340 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.460284    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22csk\" (UniqueName: \"kubernetes.io/projected/02028cc9-2ea9-437b-9d75-b6221a81ce43-kube-api-access-22csk\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.527496 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.527435    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"]
Oct 31 00:17:42.531010 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.530981    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9c21e-588d55cd7b-pk2ss"]
Oct 31 00:17:42.633756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.633717    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"]
Oct 31 00:17:42.638588 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.638559    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9c21e-747fd5b8cf-9s6gf"]
Oct 31 00:17:42.828199 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.828126    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02028cc9-2ea9-437b-9d75-b6221a81ce43" path="/var/lib/kubelet/pods/02028cc9-2ea9-437b-9d75-b6221a81ce43/volumes"
Oct 31 00:17:42.828627 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:42.828614    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c24278-cb5d-4762-8841-5c0fcf8127ad" path="/var/lib/kubelet/pods/78c24278-cb5d-4762-8841-5c0fcf8127ad/volumes"
Oct 31 00:17:44.302370 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.302325    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v" podUID="6052d389-d76d-4728-85d2-0bbf266b5bc4" containerName="payload-pre-processing" containerID="cri-o://019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b" gracePeriod=25
Oct 31 00:17:44.591101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.591070    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:17:44.708118 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.707900    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj\" (UniqueName: \"kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj\") pod \"6052d389-d76d-4728-85d2-0bbf266b5bc4\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") "
Oct 31 00:17:44.708302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.708155    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume\") pod \"6052d389-d76d-4728-85d2-0bbf266b5bc4\" (UID: \"6052d389-d76d-4728-85d2-0bbf266b5bc4\") "
Oct 31 00:17:44.709265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.709235    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume" pod "6052d389-d76d-4728-85d2-0bbf266b5bc4" (UID: "6052d389-d76d-4728-85d2-0bbf266b5bc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:44.711679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.711304    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj" pod "6052d389-d76d-4728-85d2-0bbf266b5bc4" (UID: "6052d389-d76d-4728-85d2-0bbf266b5bc4"). InnerVolumeSpecName "kube-api-access-67vvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:44.810297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.810267    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vvj\" (UniqueName: \"kubernetes.io/projected/6052d389-d76d-4728-85d2-0bbf266b5bc4-kube-api-access-67vvj\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:44.810297 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:44.810294    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052d389-d76d-4728-85d2-0bbf266b5bc4-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:45.407841 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.407799    4466 generic.go:356] "Generic (PLEG): container finished" podID="6052d389-d76d-4728-85d2-0bbf266b5bc4" containerID="019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b" exitCode=0
Oct 31 00:17:45.408347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.407871    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"
Oct 31 00:17:45.408347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.407994    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v" event={"ID":"6052d389-d76d-4728-85d2-0bbf266b5bc4","Type":"ContainerDied","Data":"019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b"}
Oct 31 00:17:45.408347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.408063    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v" event={"ID":"6052d389-d76d-4728-85d2-0bbf266b5bc4","Type":"ContainerDied","Data":"5888263fd8af4f6eb178822aa87ae90607278fc041f77fc403e765d8e3fd8882"}
Oct 31 00:17:45.408347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.408096    4466 scope.go:122] "RemoveContainer" containerID="019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b"
Oct 31 00:17:45.417827 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.417801    4466 scope.go:122] "RemoveContainer" containerID="019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b"
Oct 31 00:17:45.418739 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:17:45.418714    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b\": container with ID starting with 019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b not found: ID does not exist" containerID="019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b"
Oct 31 00:17:45.418826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.418748    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b"} err="failed to get container status \"019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b\": rpc error: code = NotFound desc = could not find container \"019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b\": container with ID starting with 019a49886778e7a04c7bbeb37cbcfc12c4d9ef5989fbc2c2be0f3bdec2e6649b not found: ID does not exist"
Oct 31 00:17:45.565231 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.565199    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"]
Oct 31 00:17:45.612080 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:45.612046    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d9e782-588c96f65f-jbm6v"]
Oct 31 00:17:46.827976 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:17:46.827943    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6052d389-d76d-4728-85d2-0bbf266b5bc4" path="/var/lib/kubelet/pods/6052d389-d76d-4728-85d2-0bbf266b5bc4/volumes"
Oct 31 00:18:08.320454 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:18:08.320416    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:18:26.170782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.170689    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"]
Oct 31 00:18:26.171379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.170977    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" podUID="e0f41986-e1cd-40b9-890b-e21f191c4736" containerName="maas-api" containerID="cri-o://2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77" gracePeriod=30
Oct 31 00:18:26.463298 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.463261    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:18:26.610966 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.609512    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\") pod \"e0f41986-e1cd-40b9-890b-e21f191c4736\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") "
Oct 31 00:18:26.610966 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.609646    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca\") pod \"e0f41986-e1cd-40b9-890b-e21f191c4736\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") "
Oct 31 00:18:26.610966 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.609740    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm\" (UniqueName: \"kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm\") pod \"e0f41986-e1cd-40b9-890b-e21f191c4736\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") "
Oct 31 00:18:26.610966 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.609829    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs\") pod \"e0f41986-e1cd-40b9-890b-e21f191c4736\" (UID: \"e0f41986-e1cd-40b9-890b-e21f191c4736\") "
Oct 31 00:18:26.625698 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.616771    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca" pod "e0f41986-e1cd-40b9-890b-e21f191c4736" (UID: "e0f41986-e1cd-40b9-890b-e21f191c4736"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:26.625698 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.624597    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs" pod "e0f41986-e1cd-40b9-890b-e21f191c4736" (UID: "e0f41986-e1cd-40b9-890b-e21f191c4736"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:26.626983 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.626942    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls" pod "e0f41986-e1cd-40b9-890b-e21f191c4736" (UID: "e0f41986-e1cd-40b9-890b-e21f191c4736"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:26.627233 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.627212    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm" pod "e0f41986-e1cd-40b9-890b-e21f191c4736" (UID: "e0f41986-e1cd-40b9-890b-e21f191c4736"). InnerVolumeSpecName "kube-api-access-b26fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:26.713485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.712294    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.713485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.712342    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f41986-e1cd-40b9-890b-e21f191c4736-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.713485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.712359    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b26fm\" (UniqueName: \"kubernetes.io/projected/e0f41986-e1cd-40b9-890b-e21f191c4736-kube-api-access-b26fm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.713485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.712376    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f41986-e1cd-40b9-890b-e21f191c4736-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:26.727234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.727201    4466 generic.go:356] "Generic (PLEG): container finished" podID="e0f41986-e1cd-40b9-890b-e21f191c4736" containerID="2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77" exitCode=0
Oct 31 00:18:26.727441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.727254    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"
Oct 31 00:18:26.727441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.727253    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" event={"ID":"e0f41986-e1cd-40b9-890b-e21f191c4736","Type":"ContainerDied","Data":"2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77"}
Oct 31 00:18:26.727441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.727381    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl" event={"ID":"e0f41986-e1cd-40b9-890b-e21f191c4736","Type":"ContainerDied","Data":"64550a6e5b040237f744f3006d301dfebbd5c1821365f802a230aff2b6bb9641"}
Oct 31 00:18:26.727441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.727411    4466 scope.go:122] "RemoveContainer" containerID="2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77"
Oct 31 00:18:26.744004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.743979    4466 scope.go:122] "RemoveContainer" containerID="2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77"
Oct 31 00:18:26.744400 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:18:26.744371    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77\": container with ID starting with 2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77 not found: ID does not exist" containerID="2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77"
Oct 31 00:18:26.744541 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.744408    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77"} err="failed to get container status \"2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77\": rpc error: code = NotFound desc = could not find container \"2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77\": container with ID starting with 2ae65df44fef54625c61238154deb6c4cc08f5688f275fefa4ae632a7a51fe77 not found: ID does not exist"
Oct 31 00:18:26.884817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.884781    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"]
Oct 31 00:18:26.887793 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:26.887769    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03c703-7bfbc8f9bb-zfvdl"]
Oct 31 00:18:28.828047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:28.827968    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f41986-e1cd-40b9-890b-e21f191c4736" path="/var/lib/kubelet/pods/e0f41986-e1cd-40b9-890b-e21f191c4736/volumes"
Oct 31 00:18:38.321002 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:18:38.320963    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:18:47.549324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:47.549288    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"]
Oct 31 00:18:52.126440 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.126398    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"]
Oct 31 00:18:52.126900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.126605    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy" containerID="cri-o://f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c" gracePeriod=30
Oct 31 00:18:52.550436 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.550323    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj" podUID="fbb0b23f-693f-4bd2-af57-e416351adf33" containerName="payload-pre-processing" containerID="cri-o://60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402" gracePeriod=25
Oct 31 00:18:52.820064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.819992    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:18:52.864126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.864088    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume\") pod \"fbb0b23f-693f-4bd2-af57-e416351adf33\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") "
Oct 31 00:18:52.864324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.864145    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b\" (UniqueName: \"kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b\") pod \"fbb0b23f-693f-4bd2-af57-e416351adf33\" (UID: \"fbb0b23f-693f-4bd2-af57-e416351adf33\") "
Oct 31 00:18:52.864475 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.864452    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume" pod "fbb0b23f-693f-4bd2-af57-e416351adf33" (UID: "fbb0b23f-693f-4bd2-af57-e416351adf33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:52.866252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.866222    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b" pod "fbb0b23f-693f-4bd2-af57-e416351adf33" (UID: "fbb0b23f-693f-4bd2-af57-e416351adf33"). InnerVolumeSpecName "kube-api-access-g5z6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:52.909731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.909691    4466 generic.go:356] "Generic (PLEG): container finished" podID="fbb0b23f-693f-4bd2-af57-e416351adf33" containerID="60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402" exitCode=0
Oct 31 00:18:52.909898 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.909795    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj" event={"ID":"fbb0b23f-693f-4bd2-af57-e416351adf33","Type":"ContainerDied","Data":"60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402"}
Oct 31 00:18:52.909898 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.909882    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj" event={"ID":"fbb0b23f-693f-4bd2-af57-e416351adf33","Type":"ContainerDied","Data":"fef974751d4b0982464de91068924d166f24851ab0e4a7d3045b7c39aedef5f5"}
Oct 31 00:18:52.910020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.909921    4466 scope.go:122] "RemoveContainer" containerID="60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402"
Oct 31 00:18:52.910339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.910123    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"
Oct 31 00:18:52.925949 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.925911    4466 scope.go:122] "RemoveContainer" containerID="60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402"
Oct 31 00:18:52.927567 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:18:52.927520    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402\": container with ID starting with 60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402 not found: ID does not exist" containerID="60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402"
Oct 31 00:18:52.927701 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.927581    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402"} err="failed to get container status \"60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402\": rpc error: code = NotFound desc = could not find container \"60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402\": container with ID starting with 60c2563cf64ff4e5c34f7ef1de6bfb0647bfbad5e3d857286fcb93d35e59b402 not found: ID does not exist"
Oct 31 00:18:52.966342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.966307    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb0b23f-693f-4bd2-af57-e416351adf33-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:52.966342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:52.966337    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5z6b\" (UniqueName: \"kubernetes.io/projected/fbb0b23f-693f-4bd2-af57-e416351adf33-kube-api-access-g5z6b\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:53.037154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:53.037121    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"]
Oct 31 00:18:53.046983 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:53.046951    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-460d4e-c97d659-rcrqj"]
Oct 31 00:18:54.855487 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:54.855383    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb0b23f-693f-4bd2-af57-e416351adf33" path="/var/lib/kubelet/pods/fbb0b23f-693f-4bd2-af57-e416351adf33/volumes"
Oct 31 00:18:57.420378 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.420351    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:18:57.532877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.532844    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.532925    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.532961    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533005    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533169 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533091    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533169 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533144    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533169 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533149    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:18:57.533321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533195    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533249    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533293    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy\") pod \"38d9550a-a812-4abc-84ff-f4a62648aaf6\" (UID: \"38d9550a-a812-4abc-84ff-f4a62648aaf6\") "
Oct 31 00:18:57.533478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533305    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:18:57.533478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533383    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:18:57.533478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533431    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:18:57.533606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533532    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:57.533927 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533901    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.534064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533929    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d9550a-a812-4abc-84ff-f4a62648aaf6-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.534064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533945    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.534064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533956    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.534064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.533964    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.535360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.535326    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:18:57.535472 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.535452    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:18:57.535735 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.535716    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "kube-api-access-d4c5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:57.535802 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.535743    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token" pod "38d9550a-a812-4abc-84ff-f4a62648aaf6" (UID: "38d9550a-a812-4abc-84ff-f4a62648aaf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:57.635237 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.635199    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.635237 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.635234    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.635466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.635247    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d9550a-a812-4abc-84ff-f4a62648aaf6-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.635466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.635259    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4c5f\" (UniqueName: \"kubernetes.io/projected/38d9550a-a812-4abc-84ff-f4a62648aaf6-kube-api-access-d4c5f\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:57.947173 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.947140    4466 generic.go:356] "Generic (PLEG): container finished" podID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerID="f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c" exitCode=0
Oct 31 00:18:57.947355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.947190    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"
Oct 31 00:18:57.947355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.947207    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" event={"ID":"38d9550a-a812-4abc-84ff-f4a62648aaf6","Type":"ContainerDied","Data":"f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c"}
Oct 31 00:18:57.947355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.947250    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc" event={"ID":"38d9550a-a812-4abc-84ff-f4a62648aaf6","Type":"ContainerDied","Data":"b5325ee24376d325ea9c74b38d4db6369c77a1b77c172f7b52d13ea2a6f74fdf"}
Oct 31 00:18:57.947355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.947271    4466 scope.go:122] "RemoveContainer" containerID="f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c"
Oct 31 00:18:57.958013 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.957997    4466 scope.go:122] "RemoveContainer" containerID="09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"
Oct 31 00:18:57.966888 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.966867    4466 scope.go:122] "RemoveContainer" containerID="f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c"
Oct 31 00:18:57.967174 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:18:57.967155    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c\": container with ID starting with f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c not found: ID does not exist" containerID="f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c"
Oct 31 00:18:57.967232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.967184    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c"} err="failed to get container status \"f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c\": rpc error: code = NotFound desc = could not find container \"f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c\": container with ID starting with f9761cca8b20c6118a47278734beb8273b7dce7f1cef02a6e3c38186b80f216c not found: ID does not exist"
Oct 31 00:18:57.967232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.967205    4466 scope.go:122] "RemoveContainer" containerID="09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"
Oct 31 00:18:57.967462 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:18:57.967439    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c\": container with ID starting with 09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c not found: ID does not exist" containerID="09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"
Oct 31 00:18:57.967525 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:57.967471    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c"} err="failed to get container status \"09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c\": rpc error: code = NotFound desc = could not find container \"09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c\": container with ID starting with 09005c93466e7f1081e4616cb0562b537364220b8005000731d7432c27a7053c not found: ID does not exist"
Oct 31 00:18:58.104495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:58.104455    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"]
Oct 31 00:18:58.151102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:58.151064    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d9e782-openshift-default-7ccffd49b5-7k8qc"]
Oct 31 00:18:58.827622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:18:58.827582    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" path="/var/lib/kubelet/pods/38d9550a-a812-4abc-84ff-f4a62648aaf6/volumes"
Oct 31 00:19:08.317567 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:08.317534    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:19:17.481204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:17.481172    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"]
Oct 31 00:19:17.481654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:17.481364    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy" containerID="cri-o://03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e" gracePeriod=30
Oct 31 00:19:17.519791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:17.519750    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"]
Oct 31 00:19:17.519979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:17.519962    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" containerID="cri-o://f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181" gracePeriod=30
Oct 31 00:19:17.528292 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:17.528259    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"]
Oct 31 00:19:17.528509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:17.528481    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy" containerID="cri-o://8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77" gracePeriod=30
Oct 31 00:19:20.283339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:20.283303    4466 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:19:20.283834 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:20.283372    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:19:22.511103 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.511073    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"]
Oct 31 00:19:22.511524 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.511383    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy" containerID="cri-o://aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1" gracePeriod=30
Oct 31 00:19:22.565051 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.565012    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"]
Oct 31 00:19:22.565335 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.565285    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" podUID="fd6de883-96fc-4b16-b843-fbb8932dfd74" containerName="maas-api" containerID="cri-o://b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4" gracePeriod=30
Oct 31 00:19:22.926567 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.926529    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:19:22.954614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.954575    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.954830 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.954729    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.954830 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.954773    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.955038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.954830    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.955038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.954929    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.955038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.954976    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.955201 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.955057    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.955201 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.955086    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.955201 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.955152    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs\") pod \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\" (UID: \"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7\") "
Oct 31 00:19:22.956077 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.956049    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.957179 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.956948    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.957668 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.957501    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "kube-api-access-42wpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:22.957668 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.957525    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:22.958424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.958003    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.958751 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.958703    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.959204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.959172    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.959419 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.959390    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:22.962107 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.962081    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token" pod "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" (UID: "5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:22.969724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.969698    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:19:22.972012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.971992    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 31 00:19:22.976304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:22.976279    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:19:23.056835 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.056747    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m\" (UniqueName: \"kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m\") pod \"fd6de883-96fc-4b16-b843-fbb8932dfd74\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") "
Oct 31 00:19:23.056835 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.056796    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.057074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.056838    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.057074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.056952    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.057074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.056990    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.057822 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.057777    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.057965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.057843    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.057965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.057880    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.057965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.057931    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.057965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.057960    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca\") pod \"fd6de883-96fc-4b16-b843-fbb8932dfd74\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") "
Oct 31 00:19:23.058164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058010    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.058164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058061    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.058164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058096    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.058164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058144    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\") pod \"fd6de883-96fc-4b16-b843-fbb8932dfd74\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") "
Oct 31 00:19:23.058377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058203    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.058377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058245    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.058377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058295    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.058377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058326    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.058377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058367    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.058621 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058415    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket\") pod \"241e425b-d2c2-4819-b417-a17a1266b094\" (UID: \"241e425b-d2c2-4819-b417-a17a1266b094\") "
Oct 31 00:19:23.058621 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058449    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs\") pod \"fd6de883-96fc-4b16-b843-fbb8932dfd74\" (UID: \"fd6de883-96fc-4b16-b843-fbb8932dfd74\") "
Oct 31 00:19:23.058621 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.058494    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs\") pod \"e73d0d2b-337d-40a3-a693-35468d86ab39\" (UID: \"e73d0d2b-337d-40a3-a693-35468d86ab39\") "
Oct 31 00:19:23.059183 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059161    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42wpd\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-kube-api-access-42wpd\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059185    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059198    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059210    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059225    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059238    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059250    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059261    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059273    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.059659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.059534    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.061597 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.061564    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:23.062994 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.062957    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.063376 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.063353    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.064525 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.064501    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m" pod "fd6de883-96fc-4b16-b843-fbb8932dfd74" (UID: "fd6de883-96fc-4b16-b843-fbb8932dfd74"). InnerVolumeSpecName "kube-api-access-hxd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.064981 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.064955    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.065325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.065287    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.068105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.067968    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.068206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068143    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.068471 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068446    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:23.068565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068463    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "kube-api-access-pvnlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.068565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068515    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.068731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068596    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls" pod "fd6de883-96fc-4b16-b843-fbb8932dfd74" (UID: "fd6de883-96fc-4b16-b843-fbb8932dfd74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:23.068785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068749    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.068785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.068752    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.069044 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.069017    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:23.069121 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.069041    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca" pod "fd6de883-96fc-4b16-b843-fbb8932dfd74" (UID: "fd6de883-96fc-4b16-b843-fbb8932dfd74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:23.069173 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.069158    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy" pod "241e425b-d2c2-4819-b417-a17a1266b094" (UID: "241e425b-d2c2-4819-b417-a17a1266b094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.070011 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.069984    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.070499 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.070477    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:23.070607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.070587    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs" pod "fd6de883-96fc-4b16-b843-fbb8932dfd74" (UID: "fd6de883-96fc-4b16-b843-fbb8932dfd74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:23.070859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.070837    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g" pod "e73d0d2b-337d-40a3-a693-35468d86ab39" (UID: "e73d0d2b-337d-40a3-a693-35468d86ab39"). InnerVolumeSpecName "kube-api-access-kkd6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.105364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.104951    4466 generic.go:356] "Generic (PLEG): container finished" podID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerID="03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e" exitCode=0
Oct 31 00:19:23.105364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.105156    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"
Oct 31 00:19:23.107234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.105837    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" event={"ID":"e73d0d2b-337d-40a3-a693-35468d86ab39","Type":"ContainerDied","Data":"03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e"}
Oct 31 00:19:23.107234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.105896    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq" event={"ID":"e73d0d2b-337d-40a3-a693-35468d86ab39","Type":"ContainerDied","Data":"f87d4f657e2e88f0635b628415a5caae47f3e71751152a1ced1f3bcc01702368"}
Oct 31 00:19:23.107234 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.105926    4466 scope.go:122] "RemoveContainer" containerID="03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e"
Oct 31 00:19:23.133675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.132765    4466 generic.go:356] "Generic (PLEG): container finished" podID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerID="8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77" exitCode=0
Oct 31 00:19:23.133675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.132877    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" event={"ID":"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7","Type":"ContainerDied","Data":"8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77"}
Oct 31 00:19:23.133675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.132929    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt" event={"ID":"5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7","Type":"ContainerDied","Data":"36d282285c28629d7776e71862bd2fb74556d2994947413f0fac4d1bc5532245"}
Oct 31 00:19:23.133675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.133005    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"
Oct 31 00:19:23.140449 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.140386    4466 generic.go:356] "Generic (PLEG): container finished" podID="241e425b-d2c2-4819-b417-a17a1266b094" containerID="f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181" exitCode=0
Oct 31 00:19:23.140585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.140522    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" event={"ID":"241e425b-d2c2-4819-b417-a17a1266b094","Type":"ContainerDied","Data":"f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181"}
Oct 31 00:19:23.140585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.140576    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z" event={"ID":"241e425b-d2c2-4819-b417-a17a1266b094","Type":"ContainerDied","Data":"0cc4290703db305be68cbee9e1fea3f82007bf22160e076e8f43e0684ed213a0"}
Oct 31 00:19:23.140727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.140699    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"
Oct 31 00:19:23.150064 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.149912    4466 scope.go:122] "RemoveContainer" containerID="03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e"
Oct 31 00:19:23.155974 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:23.155869    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e\": container with ID starting with 03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e not found: ID does not exist" containerID="03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e"
Oct 31 00:19:23.155974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.155913    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e"} err="failed to get container status \"03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e\": rpc error: code = NotFound desc = could not find container \"03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e\": container with ID starting with 03f189c9ad5e9a4008f1c3f5dbc0d3d9008701f66286709f437f16ec3924c87e not found: ID does not exist"
Oct 31 00:19:23.155974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.155938    4466 scope.go:122] "RemoveContainer" containerID="8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77"
Oct 31 00:19:23.156669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.156616    4466 generic.go:356] "Generic (PLEG): container finished" podID="fd6de883-96fc-4b16-b843-fbb8932dfd74" containerID="b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4" exitCode=0
Oct 31 00:19:23.156764 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.156710    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" event={"ID":"fd6de883-96fc-4b16-b843-fbb8932dfd74","Type":"ContainerDied","Data":"b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4"}
Oct 31 00:19:23.156764 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.156749    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj" event={"ID":"fd6de883-96fc-4b16-b843-fbb8932dfd74","Type":"ContainerDied","Data":"9e68a2c400dbda1ce151dfab36227269d9da683129dd6f892d8554fe812077fa"}
Oct 31 00:19:23.156957 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.156810    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160347    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241e425b-d2c2-4819-b417-a17a1266b094-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160374    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160390    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160405    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnlh\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-kube-api-access-pvnlh\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160421    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkd6g\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-kube-api-access-kkd6g\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160435    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241e425b-d2c2-4819-b417-a17a1266b094-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160448    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160461    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6de883-96fc-4b16-b843-fbb8932dfd74-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160475    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73d0d2b-337d-40a3-a693-35468d86ab39-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160487    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160500    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160512    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160524    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241e425b-d2c2-4819-b417-a17a1266b094-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160537    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160549    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160563    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160576    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160590    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241e425b-d2c2-4819-b417-a17a1266b094-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160605    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6de883-96fc-4b16-b843-fbb8932dfd74-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.160607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160618    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73d0d2b-337d-40a3-a693-35468d86ab39-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.161655 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160876    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxd6m\" (UniqueName: \"kubernetes.io/projected/fd6de883-96fc-4b16-b843-fbb8932dfd74-kube-api-access-hxd6m\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.161655 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.160894    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73d0d2b-337d-40a3-a693-35468d86ab39-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.170009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.169985    4466 scope.go:122] "RemoveContainer" containerID="8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77"
Oct 31 00:19:23.170326 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:23.170302    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77\": container with ID starting with 8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77 not found: ID does not exist" containerID="8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77"
Oct 31 00:19:23.170404 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.170337    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77"} err="failed to get container status \"8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77\": rpc error: code = NotFound desc = could not find container \"8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77\": container with ID starting with 8ef0d063da28e3f46e36e09fc9b1f807ce2ab80ebad02777ce00eb73cb78bd77 not found: ID does not exist"
Oct 31 00:19:23.170404 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.170363    4466 scope.go:122] "RemoveContainer" containerID="f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181"
Oct 31 00:19:23.179988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.179966    4466 scope.go:122] "RemoveContainer" containerID="492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"
Oct 31 00:19:23.189773 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.189758    4466 scope.go:122] "RemoveContainer" containerID="f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181"
Oct 31 00:19:23.190033 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:23.190013    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181\": container with ID starting with f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181 not found: ID does not exist" containerID="f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181"
Oct 31 00:19:23.190112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.190044    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181"} err="failed to get container status \"f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181\": rpc error: code = NotFound desc = could not find container \"f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181\": container with ID starting with f22b701ca7ca534a60c7396d8e860e9bc5170e5c018ae159e3b3e2fe83255181 not found: ID does not exist"
Oct 31 00:19:23.190112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.190066    4466 scope.go:122] "RemoveContainer" containerID="492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"
Oct 31 00:19:23.190295 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:23.190279    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868\": container with ID starting with 492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868 not found: ID does not exist" containerID="492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"
Oct 31 00:19:23.190360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.190301    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868"} err="failed to get container status \"492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868\": rpc error: code = NotFound desc = could not find container \"492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868\": container with ID starting with 492b2acc3a5fbb6e9c61d1a80b2c18aefcff5a300aa86eca48fad10795094868 not found: ID does not exist"
Oct 31 00:19:23.190360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.190318    4466 scope.go:122] "RemoveContainer" containerID="b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4"
Oct 31 00:19:23.201865 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.201839    4466 scope.go:122] "RemoveContainer" containerID="b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4"
Oct 31 00:19:23.202202 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:23.202175    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4\": container with ID starting with b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4 not found: ID does not exist" containerID="b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4"
Oct 31 00:19:23.202330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.202207    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4"} err="failed to get container status \"b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4\": rpc error: code = NotFound desc = could not find container \"b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4\": container with ID starting with b78d2ffb9430cf500ab3e9275b985aaf5edacf9c5c83a70cea2b826309940ea4 not found: ID does not exist"
Oct 31 00:19:23.624228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.624195    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"]
Oct 31 00:19:23.629037 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.629009    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-27e82b-cbb4f44cc-nm2bj"]
Oct 31 00:19:23.743714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.743682    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"]
Oct 31 00:19:23.748279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.748255    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-wvq8z"]
Oct 31 00:19:23.864701 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.864629    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"]
Oct 31 00:19:23.869910 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.869886    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-kl5fq"]
Oct 31 00:19:23.984942 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.984914    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"]
Oct 31 00:19:23.991113 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:23.991083    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-p8ljt"]
Oct 31 00:19:24.827376 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:24.827337    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241e425b-d2c2-4819-b417-a17a1266b094" path="/var/lib/kubelet/pods/241e425b-d2c2-4819-b417-a17a1266b094/volumes"
Oct 31 00:19:24.827914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:24.827898    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" path="/var/lib/kubelet/pods/5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7/volumes"
Oct 31 00:19:24.828309 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:24.828298    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" path="/var/lib/kubelet/pods/e73d0d2b-337d-40a3-a693-35468d86ab39/volumes"
Oct 31 00:19:24.829030 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:24.829014    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6de883-96fc-4b16-b843-fbb8932dfd74" path="/var/lib/kubelet/pods/fd6de883-96fc-4b16-b843-fbb8932dfd74/volumes"
Oct 31 00:19:27.777091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.777034    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 31 00:19:27.825312 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825286    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825321    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825363    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825413    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825453    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825469    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825577    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825586    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.825843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825616    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825679    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data\") pod \"5f335002-5dbf-48dc-8f24-0080043e516f\" (UID: \"5f335002-5dbf-48dc-8f24-0080043e516f\") "
Oct 31 00:19:27.825843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825775    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.825843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.825799    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.826132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.826017    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.826132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.826078    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:27.826243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.826166    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.826243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.826188    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.826243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.826203    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.826243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.826218    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.827881 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.827854    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.827976 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.827900    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:27.827976 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.827911    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:27.827976 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.827940    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p" pod "5f335002-5dbf-48dc-8f24-0080043e516f" (UID: "5f335002-5dbf-48dc-8f24-0080043e516f"). InnerVolumeSpecName "kube-api-access-6tc9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:27.927709 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.927680    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tc9p\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-kube-api-access-6tc9p\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.927709 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.927708    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f335002-5dbf-48dc-8f24-0080043e516f-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.927709 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.927722    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f335002-5dbf-48dc-8f24-0080043e516f-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.927959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.927736    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f335002-5dbf-48dc-8f24-0080043e516f-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.927959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:27.927747    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f335002-5dbf-48dc-8f24-0080043e516f-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:28.189272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.189237    4466 generic.go:356] "Generic (PLEG): container finished" podID="5f335002-5dbf-48dc-8f24-0080043e516f" containerID="aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1" exitCode=0
Oct 31 00:19:28.189587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.189287    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"
Oct 31 00:19:28.189587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.189325    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" event={"ID":"5f335002-5dbf-48dc-8f24-0080043e516f","Type":"ContainerDied","Data":"aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1"}
Oct 31 00:19:28.189587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.189389    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q" event={"ID":"5f335002-5dbf-48dc-8f24-0080043e516f","Type":"ContainerDied","Data":"d50c956933124493704a55c672b24f61e45b9c166d1922f258a2a5aea711928d"}
Oct 31 00:19:28.189587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.189420    4466 scope.go:122] "RemoveContainer" containerID="aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1"
Oct 31 00:19:28.199886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.199867    4466 scope.go:122] "RemoveContainer" containerID="edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"
Oct 31 00:19:28.208488 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.208466    4466 scope.go:122] "RemoveContainer" containerID="aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1"
Oct 31 00:19:28.208790 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:28.208772    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1\": container with ID starting with aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1 not found: ID does not exist" containerID="aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1"
Oct 31 00:19:28.208869 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.208800    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1"} err="failed to get container status \"aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1\": rpc error: code = NotFound desc = could not find container \"aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1\": container with ID starting with aabc94248ff89a1a1536ac9ea23396e14bb830561457c7ce5589488e215fc5a1 not found: ID does not exist"
Oct 31 00:19:28.208869 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.208818    4466 scope.go:122] "RemoveContainer" containerID="edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"
Oct 31 00:19:28.209022 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:28.209008    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157\": container with ID starting with edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157 not found: ID does not exist" containerID="edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"
Oct 31 00:19:28.209069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.209026    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157"} err="failed to get container status \"edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157\": rpc error: code = NotFound desc = could not find container \"edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157\": container with ID starting with edcd1196fac3b5f8db3f57cec0e58704af139391e52b24f9b75169da8d716157 not found: ID does not exist"
Oct 31 00:19:28.364209 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.364181    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"]
Oct 31 00:19:28.368764 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.368743    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-qc65q"]
Oct 31 00:19:28.827565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:28.827529    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" path="/var/lib/kubelet/pods/5f335002-5dbf-48dc-8f24-0080043e516f/volumes"
Oct 31 00:19:31.566051 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:31.566017    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"]
Oct 31 00:19:31.566511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:31.566232    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy" containerID="cri-o://09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3" gracePeriod=30
Oct 31 00:19:36.840161 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.840134    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 31 00:19:36.958140 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958110    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958219    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958283    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958364    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958413    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958446    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958547    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958666 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958551    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.958666 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958614    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958666 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958571    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:36.958844 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958723    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket\") pod \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\" (UID: \"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41\") "
Oct 31 00:19:36.958947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958831    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.959027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.958852    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.959027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.959012    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.959323 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.959306    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.959392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.959327    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.959392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.959337    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.959392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.959351    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.959392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.959360    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.961130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.961106    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:36.961236 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.961136    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.961236 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.961154    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "kube-api-access-5vvjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:36.961764 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:36.961743    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo" pod "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" (UID: "7bdfe64d-5157-4b10-a0c2-dc21f29c1a41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:37.061000 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.060967    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvjj\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-kube-api-access-5vvjj\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.061000 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.060997    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.061212 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.061009    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.061212 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.061017    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.244190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.244101    4466 generic.go:356] "Generic (PLEG): container finished" podID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerID="09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3" exitCode=0
Oct 31 00:19:37.244190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.244154    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"
Oct 31 00:19:37.244390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.244184    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" event={"ID":"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41","Type":"ContainerDied","Data":"09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3"}
Oct 31 00:19:37.244390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.244245    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc" event={"ID":"7bdfe64d-5157-4b10-a0c2-dc21f29c1a41","Type":"ContainerDied","Data":"0774ce36a638a4c9bb5b1705816ff4788cfff630be226fc98a2dca6ffbaa57fb"}
Oct 31 00:19:37.244390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.244275    4466 scope.go:122] "RemoveContainer" containerID="09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3"
Oct 31 00:19:37.254806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.254786    4466 scope.go:122] "RemoveContainer" containerID="09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3"
Oct 31 00:19:37.255119 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:37.255099    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3\": container with ID starting with 09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3 not found: ID does not exist" containerID="09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3"
Oct 31 00:19:37.255194 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.255128    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3"} err="failed to get container status \"09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3\": rpc error: code = NotFound desc = could not find container \"09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3\": container with ID starting with 09575a0da2a6af2063a95a2c6564f0dce5e601e865ca06ee91008b2244c483a3 not found: ID does not exist"
Oct 31 00:19:37.299462 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.299433    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"]
Oct 31 00:19:37.299695 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.299678    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" containerID="cri-o://f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c" gracePeriod=30
Oct 31 00:19:37.550485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.550396    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"]
Oct 31 00:19:37.556935 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:37.556911    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-4dqrc"]
Oct 31 00:19:38.317316 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:38.317285    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:19:38.595181 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:38.595149    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"]
Oct 31 00:19:38.595432 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:38.595399    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy" containerID="cri-o://68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611" gracePeriod=30
Oct 31 00:19:38.831439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:38.831409    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" path="/var/lib/kubelet/pods/7bdfe64d-5157-4b10-a0c2-dc21f29c1a41/volumes"
Oct 31 00:19:38.874311 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:38.874197    4466 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:19:38.874311 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:38.874292    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:19:39.573316 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:39.573274    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"]
Oct 31 00:19:39.574330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:39.573525    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" podUID="587216ab-cc26-46fa-b941-c441f1ec827e" containerName="maas-api" containerID="cri-o://5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd" gracePeriod=30
Oct 31 00:19:39.866538 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:39.866507    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:19:40.002019 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.001980    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca\") pod \"587216ab-cc26-46fa-b941-c441f1ec827e\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") "
Oct 31 00:19:40.002200 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.002069    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh\" (UniqueName: \"kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh\") pod \"587216ab-cc26-46fa-b941-c441f1ec827e\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") "
Oct 31 00:19:40.002200 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.002130    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\") pod \"587216ab-cc26-46fa-b941-c441f1ec827e\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") "
Oct 31 00:19:40.002592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.002438    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs\") pod \"587216ab-cc26-46fa-b941-c441f1ec827e\" (UID: \"587216ab-cc26-46fa-b941-c441f1ec827e\") "
Oct 31 00:19:40.002592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.002486    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca" pod "587216ab-cc26-46fa-b941-c441f1ec827e" (UID: "587216ab-cc26-46fa-b941-c441f1ec827e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:40.003649 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.002976    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587216ab-cc26-46fa-b941-c441f1ec827e-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:40.009071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.009041    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs" pod "587216ab-cc26-46fa-b941-c441f1ec827e" (UID: "587216ab-cc26-46fa-b941-c441f1ec827e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:40.009162 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.009087    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls" pod "587216ab-cc26-46fa-b941-c441f1ec827e" (UID: "587216ab-cc26-46fa-b941-c441f1ec827e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:40.009162 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.009111    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh" pod "587216ab-cc26-46fa-b941-c441f1ec827e" (UID: "587216ab-cc26-46fa-b941-c441f1ec827e"). InnerVolumeSpecName "kube-api-access-b74sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:40.104450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.104365    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:40.104450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.104396    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74sh\" (UniqueName: \"kubernetes.io/projected/587216ab-cc26-46fa-b941-c441f1ec827e-kube-api-access-b74sh\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:40.104450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.104407    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587216ab-cc26-46fa-b941-c441f1ec827e-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:40.263213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.263181    4466 generic.go:356] "Generic (PLEG): container finished" podID="587216ab-cc26-46fa-b941-c441f1ec827e" containerID="5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd" exitCode=0
Oct 31 00:19:40.263397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.263227    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"
Oct 31 00:19:40.263397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.263268    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" event={"ID":"587216ab-cc26-46fa-b941-c441f1ec827e","Type":"ContainerDied","Data":"5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd"}
Oct 31 00:19:40.263397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.263332    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8" event={"ID":"587216ab-cc26-46fa-b941-c441f1ec827e","Type":"ContainerDied","Data":"3da6bacf0dd01e17b30194e3c0410f52f4422098689f4e473357f6291b66e9fb"}
Oct 31 00:19:40.263397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.263360    4466 scope.go:122] "RemoveContainer" containerID="5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd"
Oct 31 00:19:40.273482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.273462    4466 scope.go:122] "RemoveContainer" containerID="5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd"
Oct 31 00:19:40.273847 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:40.273772    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd\": container with ID starting with 5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd not found: ID does not exist" containerID="5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd"
Oct 31 00:19:40.273847 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.273835    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd"} err="failed to get container status \"5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd\": rpc error: code = NotFound desc = could not find container \"5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd\": container with ID starting with 5f032502dc564aa0f3055b91a1b719986660bf5ef52921c4fa02085e7db75cbd not found: ID does not exist"
Oct 31 00:19:40.442566 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.442485    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"]
Oct 31 00:19:40.494590 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.494560    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-460d4e-79b8687b5d-bbln8"]
Oct 31 00:19:40.827430 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:40.827352    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587216ab-cc26-46fa-b941-c441f1ec827e" path="/var/lib/kubelet/pods/587216ab-cc26-46fa-b941-c441f1ec827e/volumes"
Oct 31 00:19:42.961921 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:42.961895    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:19:43.043240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043204    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043258    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043343    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043505 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043461    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043527    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043563    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043589    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:43.043752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043682    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:43.043752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043725    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:43.043752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043728    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043902 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043787    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.043902 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.043830    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert\") pod \"ece29ed1-b432-4c6b-9156-031a2f567516\" (UID: \"ece29ed1-b432-4c6b-9156-031a2f567516\") "
Oct 31 00:19:43.044057 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044035    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:43.044220 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044196    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:43.044450 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044434    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.044527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044451    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.044527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044462    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.044527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044474    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.044527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.044486    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece29ed1-b432-4c6b-9156-031a2f567516-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.045681 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.045662    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:43.045766 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.045735    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:43.046065 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.046044    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "kube-api-access-zbs2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:43.046132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.046048    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy" pod "ece29ed1-b432-4c6b-9156-031a2f567516" (UID: "ece29ed1-b432-4c6b-9156-031a2f567516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:43.146104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.146068    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece29ed1-b432-4c6b-9156-031a2f567516-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.146104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.146102    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece29ed1-b432-4c6b-9156-031a2f567516-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.146104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.146116    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.146428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.146130    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbs2h\" (UniqueName: \"kubernetes.io/projected/ece29ed1-b432-4c6b-9156-031a2f567516-kube-api-access-zbs2h\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:43.283178 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.283119    4466 generic.go:356] "Generic (PLEG): container finished" podID="ece29ed1-b432-4c6b-9156-031a2f567516" containerID="f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c" exitCode=0
Oct 31 00:19:43.283348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.283184    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"
Oct 31 00:19:43.283348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.283191    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" event={"ID":"ece29ed1-b432-4c6b-9156-031a2f567516","Type":"ContainerDied","Data":"f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c"}
Oct 31 00:19:43.283348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.283272    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c" event={"ID":"ece29ed1-b432-4c6b-9156-031a2f567516","Type":"ContainerDied","Data":"f8ddff1f06135a82535c5c7123e36e5440c09a2a09e2aa1703e60d69e8d763e3"}
Oct 31 00:19:43.283348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.283303    4466 scope.go:122] "RemoveContainer" containerID="f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c"
Oct 31 00:19:43.293025 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.293004    4466 scope.go:122] "RemoveContainer" containerID="f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c"
Oct 31 00:19:43.293290 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:43.293270    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c\": container with ID starting with f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c not found: ID does not exist" containerID="f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c"
Oct 31 00:19:43.293337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.293301    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c"} err="failed to get container status \"f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c\": rpc error: code = NotFound desc = could not find container \"f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c\": container with ID starting with f0393164333e97f5b6674a9b3446646256b0b89fb1193c4154355ca42c34542c not found: ID does not exist"
Oct 31 00:19:43.470082 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.470000    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"]
Oct 31 00:19:43.474161 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.474136    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-pg48c"]
Oct 31 00:19:43.905027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:43.905000    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069170    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069239    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069348    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069379    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069413    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069518    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069548    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069593    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.069992    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs\") pod \"6061f4cc-491a-493d-ab55-1836ab0b675a\" (UID: \"6061f4cc-491a-493d-ab55-1836ab0b675a\") "
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.070807    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.079714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.071522    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:44.084214 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.084169    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.084584 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.084561    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.085367 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.085344    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.087511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.086825    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.087511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.086843    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:44.088473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.088452    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "kube-api-access-kqzcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.088564 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.088471    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token" pod "6061f4cc-491a-493d-ab55-1836ab0b675a" (UID: "6061f4cc-491a-493d-ab55-1836ab0b675a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.172351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172314    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172343    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqzcw\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-kube-api-access-kqzcw\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172354    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172363    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172372    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172380    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172389    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061f4cc-491a-493d-ab55-1836ab0b675a-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172397    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061f4cc-491a-493d-ab55-1836ab0b675a-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.172578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.172405    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061f4cc-491a-493d-ab55-1836ab0b675a-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.298213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.298164    4466 generic.go:356] "Generic (PLEG): container finished" podID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerID="68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611" exitCode=0
Oct 31 00:19:44.298710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.298304    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" event={"ID":"6061f4cc-491a-493d-ab55-1836ab0b675a","Type":"ContainerDied","Data":"68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611"}
Oct 31 00:19:44.298710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.298359    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh" event={"ID":"6061f4cc-491a-493d-ab55-1836ab0b675a","Type":"ContainerDied","Data":"9e79d38d815677529d518e0d2faa7644b56c7995e080d49158a6a9b678c844a2"}
Oct 31 00:19:44.298710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.298391    4466 scope.go:122] "RemoveContainer" containerID="68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611"
Oct 31 00:19:44.298710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.298512    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"
Oct 31 00:19:44.317437 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.317416    4466 scope.go:122] "RemoveContainer" containerID="68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611"
Oct 31 00:19:44.317784 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:19:44.317759    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611\": container with ID starting with 68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611 not found: ID does not exist" containerID="68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611"
Oct 31 00:19:44.317871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.317795    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611"} err="failed to get container status \"68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611\": rpc error: code = NotFound desc = could not find container \"68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611\": container with ID starting with 68d5b228d1b3e2c4f68426c791bc8ade4472053a48ed84fd3f410bdf4dfb0611 not found: ID does not exist"
Oct 31 00:19:44.473258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.473222    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"]
Oct 31 00:19:44.477268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.477245    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-w6bmh"]
Oct 31 00:19:44.828679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.828562    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" path="/var/lib/kubelet/pods/6061f4cc-491a-493d-ab55-1836ab0b675a/volumes"
Oct 31 00:19:44.829095 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:19:44.829078    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" path="/var/lib/kubelet/pods/ece29ed1-b432-4c6b-9156-031a2f567516/volumes"
Oct 31 00:20:08.318231 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:20:08.318186    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:20:23.781078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781041    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"]
Oct 31 00:20:23.781534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781518    4466 cpu_manager.go:414] "containerMap: removing container" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy"
Oct 31 00:20:23.781576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781536    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy"
Oct 31 00:20:23.781576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781550    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy"
Oct 31 00:20:23.781576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781556    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy"
Oct 31 00:20:23.781576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781566    4466 cpu_manager.go:414] "containerMap: removing container" podUID="a47abc60-9568-43e1-82c7-e937af2ce01f" containerName="payload-processing"
Oct 31 00:20:23.781576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781571    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47abc60-9568-43e1-82c7-e937af2ce01f" containerName="payload-processing"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781580    4466 cpu_manager.go:414] "containerMap: removing container" podUID="fd6de883-96fc-4b16-b843-fbb8932dfd74" containerName="maas-api"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781584    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6de883-96fc-4b16-b843-fbb8932dfd74" containerName="maas-api"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781595    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e0f41986-e1cd-40b9-890b-e21f191c4736" containerName="maas-api"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781603    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f41986-e1cd-40b9-890b-e21f191c4736" containerName="maas-api"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781612    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6052d389-d76d-4728-85d2-0bbf266b5bc4" containerName="payload-pre-processing"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781616    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6052d389-d76d-4728-85d2-0bbf266b5bc4" containerName="payload-pre-processing"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781627    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781643    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781654    4466 cpu_manager.go:414] "containerMap: removing container" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781661    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781672    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781679    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781689    4466 cpu_manager.go:414] "containerMap: removing container" podUID="fa15182f-8783-49ce-a4e2-a32de14f12e5" containerName="payload-processing"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781695    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa15182f-8783-49ce-a4e2-a32de14f12e5" containerName="payload-processing"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781704    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781709    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781718    4466 cpu_manager.go:414] "containerMap: removing container" podUID="587216ab-cc26-46fa-b941-c441f1ec827e" containerName="maas-api"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781723    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587216ab-cc26-46fa-b941-c441f1ec827e" containerName="maas-api"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781729    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781734    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy"
Oct 31 00:20:23.781741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781744    4466 cpu_manager.go:414] "containerMap: removing container" podUID="fbb0b23f-693f-4bd2-af57-e416351adf33" containerName="payload-pre-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781750    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb0b23f-693f-4bd2-af57-e416351adf33" containerName="payload-pre-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781762    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781767    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781775    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781779    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781785    4466 cpu_manager.go:414] "containerMap: removing container" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781789    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781797    4466 cpu_manager.go:414] "containerMap: removing container" podUID="02028cc9-2ea9-437b-9d75-b6221a81ce43" containerName="payload-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781801    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02028cc9-2ea9-437b-9d75-b6221a81ce43" containerName="payload-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781807    4466 cpu_manager.go:414] "containerMap: removing container" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781812    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781818    4466 cpu_manager.go:414] "containerMap: removing container" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781822    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781832    4466 cpu_manager.go:414] "containerMap: removing container" podUID="78c24278-cb5d-4762-8841-5c0fcf8127ad" containerName="payload-pre-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781840    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c24278-cb5d-4762-8841-5c0fcf8127ad" containerName="payload-pre-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781929    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdfe64d-5157-4b10-a0c2-dc21f29c1a41" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781938    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781945    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47abc60-9568-43e1-82c7-e937af2ce01f" containerName="payload-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781951    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6de883-96fc-4b16-b843-fbb8932dfd74" containerName="maas-api"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781958    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f41986-e1cd-40b9-890b-e21f191c4736" containerName="maas-api"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781964    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6052d389-d76d-4728-85d2-0bbf266b5bc4" containerName="payload-pre-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781970    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781975    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781980    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6061f4cc-491a-493d-ab55-1836ab0b675a" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781985    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa15182f-8783-49ce-a4e2-a32de14f12e5" containerName="payload-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781990    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73d0d2b-337d-40a3-a693-35468d86ab39" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.781995    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="587216ab-cc26-46fa-b941-c441f1ec827e" containerName="maas-api"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782000    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfa9d58-8c7b-4eff-85b5-52de57f3d9c7" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782008    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb0b23f-693f-4bd2-af57-e416351adf33" containerName="payload-pre-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782017    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece29ed1-b432-4c6b-9156-031a2f567516" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782021    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782028    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="02028cc9-2ea9-437b-9d75-b6221a81ce43" containerName="payload-processing"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782033    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="241e425b-d2c2-4819-b417-a17a1266b094" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782039    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d9550a-a812-4abc-84ff-f4a62648aaf6" containerName="istio-proxy"
Oct 31 00:20:23.782364 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.782045    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c24278-cb5d-4762-8841-5c0fcf8127ad" containerName="payload-pre-processing"
Oct 31 00:20:23.784954 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.784938    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.843713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.843682    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-49837c-openshift-default-dockercfg-h8cr9\""
Oct 31 00:20:23.889226 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889187    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889302    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889346    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889387    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889441    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclf6\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889500    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889527    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889683 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889662    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.889791 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.889772    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.894975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.894949    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"]
Oct 31 00:20:23.990682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990649    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.990862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990732    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.990862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990770    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.990862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990830    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.990862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990855    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990917    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990946    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclf6\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.990983    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.991014    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991303 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.991268    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991367 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.991299    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991426 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.991395    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991533 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.991515    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.991689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.991626    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.993075 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.993046    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:23.993536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:23.993511    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:24.054877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.054802    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:24.055038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.054919    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclf6\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6\") pod \"e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") " pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:24.097770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.097736    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:24.523498 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:20:24.523451    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e949a8_6bb3_4108_818f_dc99b419fdc0.slice/crio-c2381d5b09dc8a6a33280c67e40a73e685bc67a5b6034ba5ef4894245b26e17a WatchSource:0}: Error finding container c2381d5b09dc8a6a33280c67e40a73e685bc67a5b6034ba5ef4894245b26e17a: Status 404 returned error can't find the container with id c2381d5b09dc8a6a33280c67e40a73e685bc67a5b6034ba5ef4894245b26e17a
Oct 31 00:20:24.526360 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.526317    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:20:24.526447 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.526423    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:20:24.526486 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.526465    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:20:24.527389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:24.527365    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"]
Oct 31 00:20:25.544434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:25.544396    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" event={"ID":"e5e949a8-6bb3-4108-818f-dc99b419fdc0","Type":"ContainerStarted","Data":"e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"}
Oct 31 00:20:25.544879 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:25.544477    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" event={"ID":"e5e949a8-6bb3-4108-818f-dc99b419fdc0","Type":"ContainerStarted","Data":"c2381d5b09dc8a6a33280c67e40a73e685bc67a5b6034ba5ef4894245b26e17a"}
Oct 31 00:20:25.748998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:25.748945    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podStartSLOduration=2.748930669 podStartE2EDuration="2.748930669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:25.744750755 +0000 UTC m=+2331.035299071" watchObservedRunningTime="2026-10-31 00:20:25.748930669 +0000 UTC m=+2331.039478987"
Oct 31 00:20:26.098943 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:26.098898    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:26.104808 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:26.104776    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:26.550420 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:26.550339    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:26.551201 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:26.551184    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:20:30.124750 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:30.124714    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:20:30.125143 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:30.124765    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:20:38.321073 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:20:38.320986    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:20:52.540262 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:52.540226    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"]
Oct 31 00:20:52.592600 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:52.592557    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"]
Oct 31 00:20:54.525163 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.525116    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"]
Oct 31 00:20:54.526061 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.526038    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy"
Oct 31 00:20:54.526117 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.526066    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy"
Oct 31 00:20:54.526206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.526194    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9eafdaa-1fb5-4ca0-a3b4-5c168464fac4" containerName="istio-proxy"
Oct 31 00:20:54.526248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.526212    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f335002-5dbf-48dc-8f24-0080043e516f" containerName="istio-proxy"
Oct 31 00:20:54.531449 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.531422    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.646994    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647070    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647182    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647242    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647306    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhb56\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647389    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647428    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647454    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.647682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.647602    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.662246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.662160    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"]
Oct 31 00:20:54.748985 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.748944    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749068    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749118    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749160    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749188    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhb56\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749232    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749269    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749291    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749338    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749390    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.749800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.749776    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.750042 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.750014    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.750239 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.750190    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.750818 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.750797    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.760838 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.760815    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.761997 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.761980    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.849275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.848895    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhb56\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:54.853527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:54.853485    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-49lgw\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:55.145252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.145155    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:55.581748 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:20:55.581716    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef14768b_ad29_43c8_b2a0_462e0063855a.slice/crio-b65e4a0064dc09d208f35c4797ac88861fd224177a9da801525878b707e67bde WatchSource:0}: Error finding container b65e4a0064dc09d208f35c4797ac88861fd224177a9da801525878b707e67bde: Status 404 returned error can't find the container with id b65e4a0064dc09d208f35c4797ac88861fd224177a9da801525878b707e67bde
Oct 31 00:20:55.585339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.585296    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:20:55.585465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.585407    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:20:55.585537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.585469    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:20:55.586395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.586000    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"]
Oct 31 00:20:55.764580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.764547    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" event={"ID":"ef14768b-ad29-43c8-b2a0-462e0063855a","Type":"ContainerStarted","Data":"447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd"}
Oct 31 00:20:55.764774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.764602    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" event={"ID":"ef14768b-ad29-43c8-b2a0-462e0063855a","Type":"ContainerStarted","Data":"b65e4a0064dc09d208f35c4797ac88861fd224177a9da801525878b707e67bde"}
Oct 31 00:20:55.912592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:55.912529    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podStartSLOduration=1.9125138480000001 podStartE2EDuration="1.912513848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:55.909890888 +0000 UTC m=+2361.200439206" watchObservedRunningTime="2026-10-31 00:20:55.912513848 +0000 UTC m=+2361.203062166"
Oct 31 00:20:56.149600 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:56.149571    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:20:56.153789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:56.153751    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:20:56.153989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:56.153830    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:20:57.153304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:57.152783    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:20:57.153304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:57.152858    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:20:58.149657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:58.148035    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:20:58.149657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:58.148100    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:20:59.153108 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:59.153067    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:20:59.153736 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:20:59.153701    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:21:00.152495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:00.152024    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:21:00.152495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:00.152106    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:21:01.146963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:01.146926    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:21:01.148260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:01.148230    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:21:02.146382 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:02.146339    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:21:02.146573 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:02.146413    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:21:03.146373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:03.146332    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-49lgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:21:03.148743 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:03.146414    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 00:21:04.192583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:04.192546    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:21:04.193174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:04.193003    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:21:04.194350 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:04.194326    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:21:07.560152 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.560111    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" containerID="cri-o://638bfe5cff38ad058ca4cb0d1af29aa88397b7cceb08c19f5bbeba4424a4832f" gracePeriod=45
Oct 31 00:21:07.612577 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.612533    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main" containerID="cri-o://ea10389c3c99be63c84b8a2afea0a8317d2dac9dcaf3037ee87f705d06ac1df2" gracePeriod=45
Oct 31 00:21:07.700813 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.700777    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 31 00:21:07.844367 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.844332    4466 generic.go:356] "Generic (PLEG): container finished" podID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerID="638bfe5cff38ad058ca4cb0d1af29aa88397b7cceb08c19f5bbeba4424a4832f" exitCode=0
Oct 31 00:21:07.844544 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.844403    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" event={"ID":"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa","Type":"ContainerDied","Data":"638bfe5cff38ad058ca4cb0d1af29aa88397b7cceb08c19f5bbeba4424a4832f"}
Oct 31 00:21:07.846668 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.846607    4466 generic.go:356] "Generic (PLEG): container finished" podID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerID="ea10389c3c99be63c84b8a2afea0a8317d2dac9dcaf3037ee87f705d06ac1df2" exitCode=0
Oct 31 00:21:07.847249 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.846676    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" event={"ID":"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0","Type":"ContainerDied","Data":"ea10389c3c99be63c84b8a2afea0a8317d2dac9dcaf3037ee87f705d06ac1df2"}
Oct 31 00:21:07.897441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.897411    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:21:07.900964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:07.900938    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:21:08.055242 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055198    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir\") pod \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") "
Oct 31 00:21:08.055446 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055329    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm\") pod \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") "
Oct 31 00:21:08.055446 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055398    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs\") pod \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") "
Oct 31 00:21:08.055580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055470    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home\") pod \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") "
Oct 31 00:21:08.055580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055524    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc\" (UniqueName: \"kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc\") pod \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") "
Oct 31 00:21:08.055580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055551    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs\") pod \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") "
Oct 31 00:21:08.055787 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055619    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir\") pod \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") "
Oct 31 00:21:08.055787 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055754    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm\") pod \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") "
Oct 31 00:21:08.055890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055787    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache\") pod \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\" (UID: \"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa\") "
Oct 31 00:21:08.055890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055821    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home\") pod \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") "
Oct 31 00:21:08.055890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055822    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home" pod "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" (UID: "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.055890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055870    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache\") pod \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") "
Oct 31 00:21:08.056105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055904    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst\" (UniqueName: \"kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst\") pod \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\" (UID: \"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0\") "
Oct 31 00:21:08.056105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.056016    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir" pod "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" (UID: "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.056506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.056476    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.056627 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.056511    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.056627 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.056517    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home" pod "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" (UID: "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.056763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.056727    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache" pod "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" (UID: "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.057219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.057192    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache" pod "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" (UID: "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.057346 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.055562    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir" pod "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" (UID: "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.058230 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.058194    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm" pod "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" (UID: "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.058356 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.058295    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs" pod "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" (UID: "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:08.059310 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.059275    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst" pod "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" (UID: "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0"). InnerVolumeSpecName "kube-api-access-fslst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:08.059423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.059369    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs" pod "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" (UID: "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:08.059517 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.059480    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc" pod "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" (UID: "a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa"). InnerVolumeSpecName "kube-api-access-4zxsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:08.060550 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.060526    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm" pod "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" (UID: "26f1d7e9-1e18-4204-9e6e-d86b831c4bd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:08.157359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157322    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157351    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157361    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157370    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157381    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslst\" (UniqueName: \"kubernetes.io/projected/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-kube-api-access-fslst\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157392    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157399    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157407    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157414    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxsc\" (UniqueName: \"kubernetes.io/projected/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-kube-api-access-4zxsc\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.157812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.157422    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.321172 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:21:08.321127    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:21:08.853593 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.853566    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"
Oct 31 00:21:08.853593 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.853574    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26" event={"ID":"a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa","Type":"ContainerDied","Data":"12e3f20f2fd28a88f2fa7aa56b8ed1c73316e3982663ff04fb40a88f8b1100e2"}
Oct 31 00:21:08.854151 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.853644    4466 scope.go:122] "RemoveContainer" containerID="638bfe5cff38ad058ca4cb0d1af29aa88397b7cceb08c19f5bbeba4424a4832f"
Oct 31 00:21:08.855298 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.855274    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"
Oct 31 00:21:08.855413 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.855273    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s" event={"ID":"26f1d7e9-1e18-4204-9e6e-d86b831c4bd0","Type":"ContainerDied","Data":"fc317c03d0638d36df5d63d2e25867b8c1c8ae4941800154885364da6505eaf9"}
Oct 31 00:21:08.863079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:08.863059    4466 scope.go:122] "RemoveContainer" containerID="ea10389c3c99be63c84b8a2afea0a8317d2dac9dcaf3037ee87f705d06ac1df2"
Oct 31 00:21:09.027265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:09.027223    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"]
Oct 31 00:21:09.079222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:09.079181    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/premium-w5-fe01fa-kserve-55ddb999bd-wzd2s"]
Oct 31 00:21:09.150258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:09.150218    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"]
Oct 31 00:21:09.155260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:09.155231    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/distinct-w5-fe01fa-kserve-7d4b89ffdc-4xh26"]
Oct 31 00:21:10.829424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:10.829394    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" path="/var/lib/kubelet/pods/26f1d7e9-1e18-4204-9e6e-d86b831c4bd0/volumes"
Oct 31 00:21:10.829944 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:10.829926    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" path="/var/lib/kubelet/pods/a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa/volumes"
Oct 31 00:21:18.537314 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537234    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"]
Oct 31 00:21:18.537767 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537752    4466 cpu_manager.go:414] "containerMap: removing container" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main"
Oct 31 00:21:18.537821 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537768    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main"
Oct 31 00:21:18.537821 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537785    4466 cpu_manager.go:414] "containerMap: removing container" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main"
Oct 31 00:21:18.537821 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537790    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main"
Oct 31 00:21:18.537905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537880    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bc2d0e-719a-4733-a0cd-d4cf9464c3fa" containerName="main"
Oct 31 00:21:18.537905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.537889    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f1d7e9-1e18-4204-9e6e-d86b831c4bd0" containerName="main"
Oct 31 00:21:18.542367 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.542348    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.592657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.592572    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.592851 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.592662    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.592851 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.592707    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.592851 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.592737    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.593017 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.592983    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.593074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.593014    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.593123 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.593085    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj29p\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.593213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.593134    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.593213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.593196    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.607312 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.607278    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1e5c33-openshift-default-dockercfg-pw2zn\""
Oct 31 00:21:18.677031 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.676996    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"]
Oct 31 00:21:18.702030 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.698507    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.707775    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.707814    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.707853    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.707883    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708145    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708175    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708246    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj29p\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708299    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708401    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708623    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.708869    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.709238    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.713986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.709373    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.715859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.715798    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.721218 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.721175    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.774105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.774035    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.774311 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.774120    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj29p\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p\") pod \"e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") " pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:18.859268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:18.859235    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:19.291035 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:21:19.290998    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1eed02a_eaf2_452f_8b00_a234d021547a.slice/crio-6fa22426ce10a7cfb6ae0c594c8cf11c1a114500952526d5566896cf21aa48b4 WatchSource:0}: Error finding container 6fa22426ce10a7cfb6ae0c594c8cf11c1a114500952526d5566896cf21aa48b4: Status 404 returned error can't find the container with id 6fa22426ce10a7cfb6ae0c594c8cf11c1a114500952526d5566896cf21aa48b4
Oct 31 00:21:19.292583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:19.292557    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"]
Oct 31 00:21:19.293751 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:19.293719    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:21:19.293824 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:19.293784    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:21:19.293860 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:19.293814    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:21:19.924403 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:19.924363    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" event={"ID":"e1eed02a-eaf2-452f-8b00-a234d021547a","Type":"ContainerStarted","Data":"8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"}
Oct 31 00:21:19.924926 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:19.924428    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" event={"ID":"e1eed02a-eaf2-452f-8b00-a234d021547a","Type":"ContainerStarted","Data":"6fa22426ce10a7cfb6ae0c594c8cf11c1a114500952526d5566896cf21aa48b4"}
Oct 31 00:21:20.052114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:20.052050    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podStartSLOduration=2.052036907 podStartE2EDuration="2.052036907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:20.049081814 +0000 UTC m=+2385.339630131" watchObservedRunningTime="2026-10-31 00:21:20.052036907 +0000 UTC m=+2385.342585224"
Oct 31 00:21:20.859528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:20.859479    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:20.866894 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:20.866867    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:20.930727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:20.930683    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:20.931684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:21:20.931626    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:21:38.317791 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:21:38.317743    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:22:00.042856 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:00.042824    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"]
Oct 31 00:22:00.095123 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:00.095094    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"]
Oct 31 00:22:08.317231 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:08.317190    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:22:13.959762 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:13.959692    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"]
Oct 31 00:22:13.973951 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:13.973919    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.017959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.017920    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"]
Oct 31 00:22:14.029454 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.029414    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-37abc0\""
Oct 31 00:22:14.043985 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.043947    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49zh\" (UniqueName: \"kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh\") pod \"payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.044160 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.044136    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.066557 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.066522    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-37abc0-dockercfg-hh62x\""
Oct 31 00:22:14.145963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.145896    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49zh\" (UniqueName: \"kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh\") pod \"payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.146169 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.146053    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.146908 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.146882    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.210395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.210315    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49zh\" (UniqueName: \"kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh\") pod \"payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.295287 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.295250    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:14.667957 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:22:14.667922    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a7e57a_6ccb_44ab_9cb0_301c89afbae8.slice/crio-0711b5586c82b1792d656bd069bb4d4433fb288a534dfeafd6fe79a10e6464ff WatchSource:0}: Error finding container 0711b5586c82b1792d656bd069bb4d4433fb288a534dfeafd6fe79a10e6464ff: Status 404 returned error can't find the container with id 0711b5586c82b1792d656bd069bb4d4433fb288a534dfeafd6fe79a10e6464ff
Oct 31 00:22:14.672098 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:14.672074    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"]
Oct 31 00:22:15.063562 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.063440    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" containerID="cri-o://c59b1cf66983594d2c5ebeeaf63a6cf0b04dd042bad10d5270499e21b935fa23" gracePeriod=45
Oct 31 00:22:15.114204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.114155    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" containerID="cri-o://e0791b76a0ef998e1801127767b58f367d42166a09ddc37d138276c2e1cfc19d" gracePeriod=45
Oct 31 00:22:15.142506 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.142466    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 31 00:22:15.277079 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.277043    4466 generic.go:356] "Generic (PLEG): container finished" podID="18b27e63-21b5-4321-b826-909e9d215a4d" containerID="c59b1cf66983594d2c5ebeeaf63a6cf0b04dd042bad10d5270499e21b935fa23" exitCode=0
Oct 31 00:22:15.277248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.277140    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" event={"ID":"18b27e63-21b5-4321-b826-909e9d215a4d","Type":"ContainerDied","Data":"c59b1cf66983594d2c5ebeeaf63a6cf0b04dd042bad10d5270499e21b935fa23"}
Oct 31 00:22:15.278956 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.278927    4466 generic.go:356] "Generic (PLEG): container finished" podID="663be102-4fba-4766-96e2-d2825da59efe" containerID="e0791b76a0ef998e1801127767b58f367d42166a09ddc37d138276c2e1cfc19d" exitCode=0
Oct 31 00:22:15.279094 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.278995    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" event={"ID":"663be102-4fba-4766-96e2-d2825da59efe","Type":"ContainerDied","Data":"e0791b76a0ef998e1801127767b58f367d42166a09ddc37d138276c2e1cfc19d"}
Oct 31 00:22:15.280441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.280420    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" event={"ID":"28a7e57a-6ccb-44ab-9cb0-301c89afbae8","Type":"ContainerStarted","Data":"a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a"}
Oct 31 00:22:15.280562 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.280458    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" event={"ID":"28a7e57a-6ccb-44ab-9cb0-301c89afbae8","Type":"ContainerStarted","Data":"0711b5586c82b1792d656bd069bb4d4433fb288a534dfeafd6fe79a10e6464ff"}
Oct 31 00:22:15.280657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.280564    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:15.398424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.398392    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:22:15.402551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.402526    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:22:15.406461 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.406402    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" podStartSLOduration=2.406385208 podStartE2EDuration="2.406385208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:15.403727467 +0000 UTC m=+2440.694275785" watchObservedRunningTime="2026-10-31 00:22:15.406385208 +0000 UTC m=+2440.696933526"
Oct 31 00:22:15.464014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.463980    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache\") pod \"663be102-4fba-4766-96e2-d2825da59efe\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") "
Oct 31 00:22:15.464190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464068    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir\") pod \"663be102-4fba-4766-96e2-d2825da59efe\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") "
Oct 31 00:22:15.464190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464100    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache\") pod \"18b27e63-21b5-4321-b826-909e9d215a4d\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") "
Oct 31 00:22:15.464190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464133    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir\") pod \"18b27e63-21b5-4321-b826-909e9d215a4d\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") "
Oct 31 00:22:15.464190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464158    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm\") pod \"663be102-4fba-4766-96e2-d2825da59efe\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") "
Oct 31 00:22:15.464438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464209    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs\" (UniqueName: \"kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs\") pod \"663be102-4fba-4766-96e2-d2825da59efe\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") "
Oct 31 00:22:15.464438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464247    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj\" (UniqueName: \"kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj\") pod \"18b27e63-21b5-4321-b826-909e9d215a4d\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") "
Oct 31 00:22:15.464438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464311    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62\" (UniqueName: \"kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62\") pod \"663be102-4fba-4766-96e2-d2825da59efe\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") "
Oct 31 00:22:15.464438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464361    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache" pod "663be102-4fba-4766-96e2-d2825da59efe" (UID: "663be102-4fba-4766-96e2-d2825da59efe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.464438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464372    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir" pod "663be102-4fba-4766-96e2-d2825da59efe" (UID: "663be102-4fba-4766-96e2-d2825da59efe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.464438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464396    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache" pod "18b27e63-21b5-4321-b826-909e9d215a4d" (UID: "18b27e63-21b5-4321-b826-909e9d215a4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.464761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464473    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs\") pod \"18b27e63-21b5-4321-b826-909e9d215a4d\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") "
Oct 31 00:22:15.464761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464517    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm\") pod \"18b27e63-21b5-4321-b826-909e9d215a4d\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") "
Oct 31 00:22:15.464761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464548    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home\") pod \"18b27e63-21b5-4321-b826-909e9d215a4d\" (UID: \"18b27e63-21b5-4321-b826-909e9d215a4d\") "
Oct 31 00:22:15.464761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464609    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home\") pod \"663be102-4fba-4766-96e2-d2825da59efe\" (UID: \"663be102-4fba-4766-96e2-d2825da59efe\") "
Oct 31 00:22:15.464761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464625    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir" pod "18b27e63-21b5-4321-b826-909e9d215a4d" (UID: "18b27e63-21b5-4321-b826-909e9d215a4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.465028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.464897    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home" pod "18b27e63-21b5-4321-b826-909e9d215a4d" (UID: "18b27e63-21b5-4321-b826-909e9d215a4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.465091 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465064    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home" pod "663be102-4fba-4766-96e2-d2825da59efe" (UID: "663be102-4fba-4766-96e2-d2825da59efe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.465318 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465296    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.465407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465322    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.465407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465336    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.465407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465350    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.465407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465364    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.465407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.465377    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.467001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.466975    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs" pod "18b27e63-21b5-4321-b826-909e9d215a4d" (UID: "18b27e63-21b5-4321-b826-909e9d215a4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:15.467233 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.467190    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs" pod "663be102-4fba-4766-96e2-d2825da59efe" (UID: "663be102-4fba-4766-96e2-d2825da59efe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:15.467233 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.467213    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62" pod "663be102-4fba-4766-96e2-d2825da59efe" (UID: "663be102-4fba-4766-96e2-d2825da59efe"). InnerVolumeSpecName "kube-api-access-72h62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:15.467357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.467267    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm" pod "18b27e63-21b5-4321-b826-909e9d215a4d" (UID: "18b27e63-21b5-4321-b826-909e9d215a4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.467357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.467281    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj" pod "18b27e63-21b5-4321-b826-909e9d215a4d" (UID: "18b27e63-21b5-4321-b826-909e9d215a4d"). InnerVolumeSpecName "kube-api-access-vtwsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:15.467425 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.467378    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm" pod "663be102-4fba-4766-96e2-d2825da59efe" (UID: "663be102-4fba-4766-96e2-d2825da59efe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.567797 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.567749    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/663be102-4fba-4766-96e2-d2825da59efe-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.567797 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.567796    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwsj\" (UniqueName: \"kubernetes.io/projected/18b27e63-21b5-4321-b826-909e9d215a4d-kube-api-access-vtwsj\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.568009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.567812    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72h62\" (UniqueName: \"kubernetes.io/projected/663be102-4fba-4766-96e2-d2825da59efe-kube-api-access-72h62\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.568009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.567827    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b27e63-21b5-4321-b826-909e9d215a4d-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.568009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.567843    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b27e63-21b5-4321-b826-909e9d215a4d-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.568009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:15.567858    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/663be102-4fba-4766-96e2-d2825da59efe-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:16.287931 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.287892    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"
Oct 31 00:22:16.287931 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.287919    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds" event={"ID":"18b27e63-21b5-4321-b826-909e9d215a4d","Type":"ContainerDied","Data":"8970739d310fd61929d0fe810652267fe5a7ac07da555177808f1b0cc38893d4"}
Oct 31 00:22:16.288470 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.287986    4466 scope.go:122] "RemoveContainer" containerID="c59b1cf66983594d2c5ebeeaf63a6cf0b04dd042bad10d5270499e21b935fa23"
Oct 31 00:22:16.291223 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.291196    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"
Oct 31 00:22:16.291352 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.291237    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" event={"ID":"663be102-4fba-4766-96e2-d2825da59efe","Type":"ContainerDied","Data":"a815d2f8efc48491a1339975d24319c3fa44d212a3c56de66c2bfc641012e89a"}
Oct 31 00:22:16.304648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.304600    4466 scope.go:122] "RemoveContainer" containerID="e0791b76a0ef998e1801127767b58f367d42166a09ddc37d138276c2e1cfc19d"
Oct 31 00:22:16.393390 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.393352    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:16.461222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.461190    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"]
Oct 31 00:22:16.506947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.506908    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/simulator-w6-35a187-kserve-6dc8489dd8-ljlwn"]
Oct 31 00:22:16.623014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.622974    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"]
Oct 31 00:22:16.627045 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.627017    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/distinct-w6-35a187-kserve-846b7dc696-rq8ds"]
Oct 31 00:22:16.834991 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.834956    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" path="/var/lib/kubelet/pods/18b27e63-21b5-4321-b826-909e9d215a4d/volumes"
Oct 31 00:22:16.835423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:16.835410    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663be102-4fba-4766-96e2-d2825da59efe" path="/var/lib/kubelet/pods/663be102-4fba-4766-96e2-d2825da59efe/volumes"
Oct 31 00:22:21.758750 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:21.758703    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"]
Oct 31 00:22:21.759196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:21.758928    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" podUID="b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" containerName="maas-api" containerID="cri-o://883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec" gracePeriod=30
Oct 31 00:22:22.182036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.181996    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 31 00:22:22.275565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.275525    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8\" (UniqueName: \"kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8\") pod \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") "
Oct 31 00:22:22.275805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.275582    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca\") pod \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") "
Oct 31 00:22:22.275805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.275734    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\") pod \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") "
Oct 31 00:22:22.275805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.275755    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs\") pod \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\" (UID: \"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83\") "
Oct 31 00:22:22.275990 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.275929    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca" pod "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" (UID: "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:22.276166 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.276150    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:22.277855 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.277826    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs" pod "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" (UID: "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:22.277855 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.277845    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls" pod "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" (UID: "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:22.278008 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.277916    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8" pod "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" (UID: "b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83"). InnerVolumeSpecName "kube-api-access-hvjm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:22.343660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.332669    4466 generic.go:356] "Generic (PLEG): container finished" podID="b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" containerID="883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec" exitCode=0
Oct 31 00:22:22.343660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.332753    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" event={"ID":"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83","Type":"ContainerDied","Data":"883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec"}
Oct 31 00:22:22.343660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.332844    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn" event={"ID":"b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83","Type":"ContainerDied","Data":"9f05963928eb668508354dd485b544eb311285d2448462533155539cbfe9a9ea"}
Oct 31 00:22:22.343660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.332873    4466 scope.go:122] "RemoveContainer" containerID="883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec"
Oct 31 00:22:22.343660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.333022    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"
Oct 31 00:22:22.363357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.363333    4466 scope.go:122] "RemoveContainer" containerID="883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec"
Oct 31 00:22:22.363777 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:22.363754    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec\": container with ID starting with 883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec not found: ID does not exist" containerID="883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec"
Oct 31 00:22:22.363905 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.363883    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec"} err="failed to get container status \"883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec\": rpc error: code = NotFound desc = could not find container \"883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec\": container with ID starting with 883ae4c515d279d91127e164b5381bcdebeedfd25f80bd5116bff9e990757fec not found: ID does not exist"
Oct 31 00:22:22.377301 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.377265    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvjm8\" (UniqueName: \"kubernetes.io/projected/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-kube-api-access-hvjm8\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:22.377301 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.377298    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:22.377301 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.377308    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:22.518877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.518838    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"]
Oct 31 00:22:22.524691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.524658    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe01fa-6f479d7655-g98rn"]
Oct 31 00:22:22.831733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:22.829702    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" path="/var/lib/kubelet/pods/b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83/volumes"
Oct 31 00:22:26.294370 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:26.294330    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:22:27.592816 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:27.592773    4466 patch_prober.go:29] interesting pod/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:22:27.593405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:27.592895    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:22:30.404168 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:30.404131    4466 generic.go:356] "Generic (PLEG): container finished" podID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerID="e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527" exitCode=0
Oct 31 00:22:30.404672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:30.404183    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" event={"ID":"e5e949a8-6bb3-4108-818f-dc99b419fdc0","Type":"ContainerDied","Data":"e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"}
Oct 31 00:22:30.404672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:30.404647    4466 scope.go:122] "RemoveContainer" containerID="e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"
Oct 31 00:22:30.406258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:30.406220    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:22:30.406350 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:30.406322    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:22:30.406396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:30.406362    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:22:31.369156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:31.369118    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"]
Oct 31 00:22:31.369435 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:31.369405    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy" containerID="cri-o://186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1" gracePeriod=30
Oct 31 00:22:31.411649 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:31.411603    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" event={"ID":"e5e949a8-6bb3-4108-818f-dc99b419fdc0","Type":"ContainerStarted","Data":"f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978"}
Oct 31 00:22:32.098979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:32.098935    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:22:32.099246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:32.099221    4466 patch_prober.go:29] interesting pod/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:22:32.099330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:32.099280    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 00:22:33.098508 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.098470    4466 patch_prober.go:29] interesting pod/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:22:33.099014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.098527    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 00:22:33.196556 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.196523    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"]
Oct 31 00:22:33.197033 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197020    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" containerName="maas-api"
Oct 31 00:22:33.197084 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197034    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" containerName="maas-api"
Oct 31 00:22:33.197084 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197045    4466 cpu_manager.go:414] "containerMap: removing container" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main"
Oct 31 00:22:33.197084 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197050    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main"
Oct 31 00:22:33.197084 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197058    4466 cpu_manager.go:414] "containerMap: removing container" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main"
Oct 31 00:22:33.197084 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197064    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main"
Oct 31 00:22:33.197239 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197152    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ac7d2a-d3a6-4f0d-b75f-a778f5e67f83" containerName="maas-api"
Oct 31 00:22:33.197239 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197162    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="663be102-4fba-4766-96e2-d2825da59efe" containerName="main"
Oct 31 00:22:33.197239 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.197169    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b27e63-21b5-4321-b826-909e9d215a4d" containerName="main"
Oct 31 00:22:33.200373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.200351    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.257338 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.257304    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"]
Oct 31 00:22:33.269661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.269184    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1e5c33-serving-cert\""
Oct 31 00:22:33.375617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.375533    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.375792 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.375766    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.375835 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.375796    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692xp\" (UniqueName: \"kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.375835 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.375830    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.478975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.477607    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.478975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.477879    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.478975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.477926    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692xp\" (UniqueName: \"kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.478975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.477970    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.478975 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:33.478148    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-serving-cert: secret "maas-api-e2e-api-b-1e5c33-serving-cert" not found
Oct 31 00:22:33.478975 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:33.478281    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls podName:aaae1956-e766-462b-b8ae-20fd73f7f48c nodeName:}" failed. No retries permitted until 2026-10-31 00:22:33.978240684 +0000 UTC m=+2459.268788992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls") pod "maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" (UID: "aaae1956-e766-462b-b8ae-20fd73f7f48c") : secret "maas-api-e2e-api-b-1e5c33-serving-cert" not found
Oct 31 00:22:33.482180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.481402    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.484686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.483188    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.554548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.554507    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692xp\" (UniqueName: \"kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.986661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.986610    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:33.989028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:33.989008    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\") pod \"maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:34.099032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:34.098995    4466 patch_prober.go:29] interesting pod/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:22:34.099411 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:34.099052    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 00:22:34.115999 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:34.115956    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:34.520542 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:22:34.520509    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaae1956_e766_462b_b8ae_20fd73f7f48c.slice/crio-6099956ac24489a229e445b09a931044abb1c4b68f8274de8bb2558d90705153 WatchSource:0}: Error finding container 6099956ac24489a229e445b09a931044abb1c4b68f8274de8bb2558d90705153: Status 404 returned error can't find the container with id 6099956ac24489a229e445b09a931044abb1c4b68f8274de8bb2558d90705153
Oct 31 00:22:34.522257 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:34.522221    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:22:34.523489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:34.523468    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"]
Oct 31 00:22:35.098193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:35.098159    4466 patch_prober.go:29] interesting pod/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:22:35.098476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:35.098447    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 00:22:35.462025 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:35.461937    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" event={"ID":"aaae1956-e766-462b-b8ae-20fd73f7f48c","Type":"ContainerStarted","Data":"6099956ac24489a229e445b09a931044abb1c4b68f8274de8bb2558d90705153"}
Oct 31 00:22:36.098316 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.098260    4466 patch_prober.go:29] interesting pod/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:22:36.098500 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.098333    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 00:22:36.469726 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.469687    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" event={"ID":"aaae1956-e766-462b-b8ae-20fd73f7f48c","Type":"ContainerStarted","Data":"d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2"}
Oct 31 00:22:36.470162 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.469785    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:22:36.754623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.754593    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 31 00:22:36.838711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.838666    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.838906 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.838739    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.838906 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.838804    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.838906 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.838874    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.839072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.838925    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.839072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839000    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.839072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839047    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.839228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839140    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.839228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839182    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket\") pod \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\" (UID: \"9e83d72a-ac30-48b1-86f1-83b0f29514f7\") "
Oct 31 00:22:36.839228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839178    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.839228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839194    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:36.839530 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839461    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.839708 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839511    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.839806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839573    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.839930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839906    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.839996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839935    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.839996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839951    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.839996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.839966    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.841197 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.841165    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.841283 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.841172    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:36.841283 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.841199    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:22:36.841356 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.841285    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4" pod "9e83d72a-ac30-48b1-86f1-83b0f29514f7" (UID: "9e83d72a-ac30-48b1-86f1-83b0f29514f7"). InnerVolumeSpecName "kube-api-access-jqkh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:36.881314 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.881251    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" podStartSLOduration=2.9414918119999998 podStartE2EDuration="3.881233152s" totalImagesPullingTime="939.74134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:36.597863783 +0000 UTC m=+2461.888412097" watchObservedRunningTime="2026-10-31 00:22:36.881233152 +0000 UTC m=+2462.171781469"
Oct 31 00:22:36.941326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.941282    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.941326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.941316    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.941326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.941326    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.941326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.941334    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqkh4\" (UniqueName: \"kubernetes.io/projected/9e83d72a-ac30-48b1-86f1-83b0f29514f7-kube-api-access-jqkh4\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.941663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:36.941345    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e83d72a-ac30-48b1-86f1-83b0f29514f7-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:37.104367 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.104334    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:22:37.104776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.104748    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:22:37.105663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.105626    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:22:37.477716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.477610    4466 generic.go:356] "Generic (PLEG): container finished" podID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerID="186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1" exitCode=0
Oct 31 00:22:37.477716 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.477699    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"
Oct 31 00:22:37.478213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.477693    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" event={"ID":"9e83d72a-ac30-48b1-86f1-83b0f29514f7","Type":"ContainerDied","Data":"186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1"}
Oct 31 00:22:37.478213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.477814    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85" event={"ID":"9e83d72a-ac30-48b1-86f1-83b0f29514f7","Type":"ContainerDied","Data":"e3b3b51bbb663275cd53c0e55342412c6346c3135f73cdd78013c82e7dd0792f"}
Oct 31 00:22:37.478213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.477840    4466 scope.go:122] "RemoveContainer" containerID="186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1"
Oct 31 00:22:37.488548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.488434    4466 scope.go:122] "RemoveContainer" containerID="888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"
Oct 31 00:22:37.497273 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.497253    4466 scope.go:122] "RemoveContainer" containerID="186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1"
Oct 31 00:22:37.497567 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:37.497548    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1\": container with ID starting with 186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1 not found: ID does not exist" containerID="186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1"
Oct 31 00:22:37.497604 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.497581    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1"} err="failed to get container status \"186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1\": rpc error: code = NotFound desc = could not find container \"186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1\": container with ID starting with 186b02ab76f00b50cab995aaca036c658de19e4b8bc54cceb3bcc57b9c8f1dd1 not found: ID does not exist"
Oct 31 00:22:37.497604 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.497599    4466 scope.go:122] "RemoveContainer" containerID="888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"
Oct 31 00:22:37.497881 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:37.497864    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab\": container with ID starting with 888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab not found: ID does not exist" containerID="888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"
Oct 31 00:22:37.497917 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.497887    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab"} err="failed to get container status \"888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab\": rpc error: code = NotFound desc = could not find container \"888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab\": container with ID starting with 888a1fd1b2a08755d1d4a5c49f959ec9391ab514bd82b9f632a15a81eb793dab not found: ID does not exist"
Oct 31 00:22:37.651398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.651357    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"]
Oct 31 00:22:37.657097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:37.657068    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-2mn85"]
Oct 31 00:22:38.370488 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:22:38.370442    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:22:38.826997 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:38.826919    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" path="/var/lib/kubelet/pods/9e83d72a-ac30-48b1-86f1-83b0f29514f7/volumes"
Oct 31 00:22:42.484396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:22:42.484366    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:23:01.625373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:01.625336    4466 generic.go:356] "Generic (PLEG): container finished" podID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerID="8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455" exitCode=0
Oct 31 00:23:01.625890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:01.625411    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" event={"ID":"e1eed02a-eaf2-452f-8b00-a234d021547a","Type":"ContainerDied","Data":"8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"}
Oct 31 00:23:01.625932 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:01.625915    4466 scope.go:122] "RemoveContainer" containerID="8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"
Oct 31 00:23:01.627647 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:01.627600    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:23:01.627781 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:01.627719    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:23:01.627781 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:01.627766    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:23:02.635964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:02.635925    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" event={"ID":"e1eed02a-eaf2-452f-8b00-a234d021547a","Type":"ContainerStarted","Data":"afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081"}
Oct 31 00:23:02.859968 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:02.859921    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:23:02.861133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:02.861105    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:23:02.861265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:02.861153    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 00:23:03.859700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:03.859627    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:23:03.860132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:03.859726    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 00:23:04.859786 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:04.859746    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:23:04.860220 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:04.859829    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 00:23:05.860346 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:05.860292    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:23:05.860842 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:05.860358    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 00:23:05.947659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:05.935755    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:23:06.860072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:06.860036    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:23:06.860265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:06.860090    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 00:23:07.860219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:07.860179    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:23:07.860756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:07.860300    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 00:23:08.319062 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:23:08.318977    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:23:08.864434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:08.864402    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:23:08.865758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:08.865736    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:23:19.404190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:19.404153    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"]
Oct 31 00:23:19.404629 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:19.404370    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" containerID="cri-o://6ba01b36ea73a9fe73fe86ef2400feb5ad6f26b9d676c71b6aa06055aeb9110e" gracePeriod=30
Oct 31 00:23:23.620131 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:23.620086    4466 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:23:23.620538 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:23.620163    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:23:24.791812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:24.791725    4466 generic.go:356] "Generic (PLEG): container finished" podID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerID="6ba01b36ea73a9fe73fe86ef2400feb5ad6f26b9d676c71b6aa06055aeb9110e" exitCode=0
Oct 31 00:23:24.792205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:24.791799    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" event={"ID":"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5","Type":"ContainerDied","Data":"6ba01b36ea73a9fe73fe86ef2400feb5ad6f26b9d676c71b6aa06055aeb9110e"}
Oct 31 00:23:24.792205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:24.791896    4466 scope.go:122] "RemoveContainer" containerID="311993b736c51d5b6e46ee8f85496f63e80df0a0dd5bee8c00208098fecf23b2"
Oct 31 00:23:25.109424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.109400    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 31 00:23:25.150261 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150220    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150304    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150392    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150469    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150496    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150572    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:25.150680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150581    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150624    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150668    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150946 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150702    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:25.150946 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150737    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy\") pod \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\" (UID: \"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5\") "
Oct 31 00:23:25.150946 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.150919    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151147    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151179    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151414    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151438    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151453    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151467    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.151516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.151482    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.152815 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.152784    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "kube-api-access-vqxhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:25.152815 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.152801    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:25.152996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.152850    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:23:25.152996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.152988    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy" pod "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" (UID: "5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:25.252686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.252617    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.252686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.252679    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqxhs\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-kube-api-access-vqxhs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.252686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.252696    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.252935 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.252705    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:25.788293 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.788259    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"]
Oct 31 00:23:25.788534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.788462    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r" podUID="dc82656a-5038-49ec-8910-82069ea2905a" containerName="maas-api" containerID="cri-o://cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520" gracePeriod=30
Oct 31 00:23:25.800690 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.800649    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj" event={"ID":"5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5","Type":"ContainerDied","Data":"1cd6a4f826684b730e46d6fef3306c5492ca56ce453d83a0a77e79f2efd6b47b"}
Oct 31 00:23:25.800690 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.800678    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"
Oct 31 00:23:25.801139 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:25.800728    4466 scope.go:122] "RemoveContainer" containerID="6ba01b36ea73a9fe73fe86ef2400feb5ad6f26b9d676c71b6aa06055aeb9110e"
Oct 31 00:23:26.036087 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.036053    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"]
Oct 31 00:23:26.047226 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.047134    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-ggkpj"]
Oct 31 00:23:26.092358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.092326    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 31 00:23:26.166456 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.166413    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca\") pod \"dc82656a-5038-49ec-8910-82069ea2905a\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") "
Oct 31 00:23:26.166710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.166518    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6\" (UniqueName: \"kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6\") pod \"dc82656a-5038-49ec-8910-82069ea2905a\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") "
Oct 31 00:23:26.166710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.166573    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs\") pod \"dc82656a-5038-49ec-8910-82069ea2905a\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") "
Oct 31 00:23:26.166710 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.166605    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls\") pod \"dc82656a-5038-49ec-8910-82069ea2905a\" (UID: \"dc82656a-5038-49ec-8910-82069ea2905a\") "
Oct 31 00:23:26.168062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.168026    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca" pod "dc82656a-5038-49ec-8910-82069ea2905a" (UID: "dc82656a-5038-49ec-8910-82069ea2905a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:23:26.176687 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.171403    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6" pod "dc82656a-5038-49ec-8910-82069ea2905a" (UID: "dc82656a-5038-49ec-8910-82069ea2905a"). InnerVolumeSpecName "kube-api-access-nqzq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:26.184833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.184777    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls" pod "dc82656a-5038-49ec-8910-82069ea2905a" (UID: "dc82656a-5038-49ec-8910-82069ea2905a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:23:26.184996 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.184872    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs" pod "dc82656a-5038-49ec-8910-82069ea2905a" (UID: "dc82656a-5038-49ec-8910-82069ea2905a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:23:26.268849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.268809    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:26.268849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.268838    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc82656a-5038-49ec-8910-82069ea2905a-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:26.268849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.268850    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc82656a-5038-49ec-8910-82069ea2905a-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:26.269087 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.268858    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqzq6\" (UniqueName: \"kubernetes.io/projected/dc82656a-5038-49ec-8910-82069ea2905a-kube-api-access-nqzq6\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:26.808262 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.808220    4466 generic.go:356] "Generic (PLEG): container finished" podID="dc82656a-5038-49ec-8910-82069ea2905a" containerID="cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520" exitCode=0
Oct 31 00:23:26.808727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.808278    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"
Oct 31 00:23:26.808727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.808340    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r" event={"ID":"dc82656a-5038-49ec-8910-82069ea2905a","Type":"ContainerDied","Data":"cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520"}
Oct 31 00:23:26.808727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.808418    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r" event={"ID":"dc82656a-5038-49ec-8910-82069ea2905a","Type":"ContainerDied","Data":"d4423fc940f5560063e8ac99c595618df9f8e7e476d7d97489284a76c1d7ae89"}
Oct 31 00:23:26.808727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.808444    4466 scope.go:122] "RemoveContainer" containerID="cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520"
Oct 31 00:23:26.821789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.821770    4466 scope.go:122] "RemoveContainer" containerID="cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520"
Oct 31 00:23:26.822167 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:23:26.822145    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520\": container with ID starting with cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520 not found: ID does not exist" containerID="cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520"
Oct 31 00:23:26.822293 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.822177    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520"} err="failed to get container status \"cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520\": rpc error: code = NotFound desc = could not find container \"cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520\": container with ID starting with cc4ab3db0a9e37fd715f9cdf5bc91b6a05887ba916e35ce8dbc812b242940520 not found: ID does not exist"
Oct 31 00:23:26.828408 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.828381    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" path="/var/lib/kubelet/pods/5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5/volumes"
Oct 31 00:23:26.829337 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:23:26.829313    4466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc82656a_5038_49ec_8910_82069ea2905a.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:23:26.980901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.980868    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"]
Oct 31 00:23:26.986112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:26.986075    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-35a187-7d779745b4-m9x8r"]
Oct 31 00:23:28.827056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:28.826970    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc82656a-5038-49ec-8910-82069ea2905a" path="/var/lib/kubelet/pods/dc82656a-5038-49ec-8910-82069ea2905a/volumes"
Oct 31 00:23:35.377046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:35.377004    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"]
Oct 31 00:23:35.377654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:35.377267    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy" containerID="cri-o://ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099" gracePeriod=30
Oct 31 00:23:38.320407 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:23:38.320365    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:23:40.647453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.647383    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 31 00:23:40.761206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761174    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761250    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761288    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761357    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761410 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761399    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761630 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761435    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761630 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761516    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.761630 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761541    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761630 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761573    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.761630 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761615    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761881 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761671    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token\") pod \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\" (UID: \"cf9fdfb5-5853-495b-9a47-658ce9ba9861\") "
Oct 31 00:23:40.761881 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761737    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.761974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.761883    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.762054 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.762034    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:23:40.762359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.762339    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.762458 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.762362    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.762458 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.762376    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.762458 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.762390    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.762458 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.762402    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.763988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.763963    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:40.764098 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.764078    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.764222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.764203    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:23:40.764333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.764316    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl" pod "cf9fdfb5-5853-495b-9a47-658ce9ba9861" (UID: "cf9fdfb5-5853-495b-9a47-658ce9ba9861"). InnerVolumeSpecName "kube-api-access-pzqsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:40.863587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.863538    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.863587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.863573    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqsl\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-kube-api-access-pzqsl\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.863587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.863586    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.863587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.863595    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9fdfb5-5853-495b-9a47-658ce9ba9861-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.888761 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.888724    4466 generic.go:356] "Generic (PLEG): container finished" podID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerID="ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099" exitCode=0
Oct 31 00:23:40.888953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.888771    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"
Oct 31 00:23:40.888953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.888824    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" event={"ID":"cf9fdfb5-5853-495b-9a47-658ce9ba9861","Type":"ContainerDied","Data":"ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099"}
Oct 31 00:23:40.888953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.888877    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr" event={"ID":"cf9fdfb5-5853-495b-9a47-658ce9ba9861","Type":"ContainerDied","Data":"b9465baca19ed973aa61c053d0124e47d9fa4d7399bed8ccee325ab97740358b"}
Oct 31 00:23:40.888953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.888900    4466 scope.go:122] "RemoveContainer" containerID="ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099"
Oct 31 00:23:40.899228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.899206    4466 scope.go:122] "RemoveContainer" containerID="ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099"
Oct 31 00:23:40.899596 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:23:40.899572    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099\": container with ID starting with ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099 not found: ID does not exist" containerID="ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099"
Oct 31 00:23:40.899734 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:40.899605    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099"} err="failed to get container status \"ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099\": rpc error: code = NotFound desc = could not find container \"ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099\": container with ID starting with ca8cc5b2fca24d245d2aaf3628f5094271dbf2a289c96099fd084a17e1cbc099 not found: ID does not exist"
Oct 31 00:23:41.063581 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:41.063545    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"]
Oct 31 00:23:41.067032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:41.067009    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-657lr"]
Oct 31 00:23:42.827090 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:42.827059    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" path="/var/lib/kubelet/pods/cf9fdfb5-5853-495b-9a47-658ce9ba9861/volumes"
Oct 31 00:23:58.491345 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491303    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"]
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491815    4466 cpu_manager.go:414] "containerMap: removing container" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy"
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491828    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy"
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491843    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy"
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491848    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy"
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491855    4466 cpu_manager.go:414] "containerMap: removing container" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy"
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491861    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy"
Oct 31 00:23:58.491874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491877    4466 cpu_manager.go:414] "containerMap: removing container" podUID="dc82656a-5038-49ec-8910-82069ea2905a" containerName="maas-api"
Oct 31 00:23:58.492120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491882    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc82656a-5038-49ec-8910-82069ea2905a" containerName="maas-api"
Oct 31 00:23:58.492120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491975    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy"
Oct 31 00:23:58.492120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491982    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy"
Oct 31 00:23:58.492120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491988    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc82656a-5038-49ec-8910-82069ea2905a" containerName="maas-api"
Oct 31 00:23:58.492120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.491999    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy"
Oct 31 00:23:58.492120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.492005    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9fdfb5-5853-495b-9a47-658ce9ba9861" containerName="istio-proxy"
Oct 31 00:23:58.496819 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.496796    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.551162 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.551091    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"]
Oct 31 00:23:58.599806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.599768    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxm4f\" (UniqueName: \"kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.600081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.599965    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.600081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.600058    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.600231 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.600204    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:23:58.600273 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.600227    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:23:58.701256 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.701221    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.701468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.701289    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.701468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.701375    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxm4f\" (UniqueName: \"kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.701944 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.701915    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.703833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.703809    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.764913 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.764842    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxm4f\" (UniqueName: \"kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f\") pod \"maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:58.810777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:58.810739    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:23:59.181749 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:23:59.181704    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979a3036_14ca_45ce_bdaa_0c6e6b5bfb1e.slice/crio-e2a9582491cb12bd1dc2c13da22ce69cb948fae52c3038ca69126408ae83e22f WatchSource:0}: Error finding container e2a9582491cb12bd1dc2c13da22ce69cb948fae52c3038ca69126408ae83e22f: Status 404 returned error can't find the container with id e2a9582491cb12bd1dc2c13da22ce69cb948fae52c3038ca69126408ae83e22f
Oct 31 00:23:59.185445 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:59.185419    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"]
Oct 31 00:23:59.998157 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:59.998108    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt" event={"ID":"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e","Type":"ContainerStarted","Data":"20a291f1f4c5a5d9f3c7a0a7a4b07aef0501e5fcd41c5679a55247428591635d"}
Oct 31 00:23:59.998720 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:23:59.998186    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt" event={"ID":"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e","Type":"ContainerStarted","Data":"e2a9582491cb12bd1dc2c13da22ce69cb948fae52c3038ca69126408ae83e22f"}
Oct 31 00:24:00.150347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:00.150257    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt" podStartSLOduration=2.150242697 podStartE2EDuration="2.150242697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:00.148099382 +0000 UTC m=+2545.438647700" watchObservedRunningTime="2026-10-31 00:24:00.150242697 +0000 UTC m=+2545.440791009"
Oct 31 00:24:01.004850 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:01.004813    4466 generic.go:356] "Generic (PLEG): container finished" podID="979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" containerID="20a291f1f4c5a5d9f3c7a0a7a4b07aef0501e5fcd41c5679a55247428591635d" exitCode=0
Oct 31 00:24:01.005263 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:01.004867    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt" event={"ID":"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e","Type":"ContainerDied","Data":"20a291f1f4c5a5d9f3c7a0a7a4b07aef0501e5fcd41c5679a55247428591635d"}
Oct 31 00:24:02.160255 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.160224    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:24:02.248828 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.248786    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca\") pod \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") "
Oct 31 00:24:02.249043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.248899    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f\" (UniqueName: \"kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f\") pod \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") "
Oct 31 00:24:02.249043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.248953    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca\") pod \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\" (UID: \"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e\") "
Oct 31 00:24:02.249336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.249304    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca" pod "979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" (UID: "979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:02.249558 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.249534    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-openshift-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:02.251147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.251122    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f" pod "979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" (UID: "979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e"). InnerVolumeSpecName "kube-api-access-vxm4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:02.251236 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.251160    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca" pod "979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" (UID: "979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:24:02.350473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.350431    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxm4f\" (UniqueName: \"kubernetes.io/projected/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-kube-api-access-vxm4f\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:02.350473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:02.350466    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e-maas-api-xks-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:24:03.017705 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:03.017673    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"
Oct 31 00:24:03.017879 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:03.017666    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt" event={"ID":"979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e","Type":"ContainerDied","Data":"e2a9582491cb12bd1dc2c13da22ce69cb948fae52c3038ca69126408ae83e22f"}
Oct 31 00:24:03.017879 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:24:03.017791    4466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a9582491cb12bd1dc2c13da22ce69cb948fae52c3038ca69126408ae83e22f"
Oct 31 00:24:08.318371 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:24:08.318330    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:24:38.318889 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:24:38.318843    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:25:08.317335 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:25:08.317294    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:25:38.318144 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:25:38.318105    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:25:46.164277 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:25:46.164239    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"]
Oct 31 00:25:46.216857 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:25:46.216822    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae3cd0-nhkjt"]
Oct 31 00:25:46.827255 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:25:46.827219    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" path="/var/lib/kubelet/pods/979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e/volumes"
Oct 31 00:26:05.741501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:05.741468    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"]
Oct 31 00:26:05.741993 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:05.741710    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" containerName="discovery" containerID="cri-o://d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc" gracePeriod=30
Oct 31 00:26:06.033499 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.033458    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:26:06.101302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101265    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.101490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101354    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.101490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101397    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.101490 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101443    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.101607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101590    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.101712 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101687    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.101924 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101895    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:06.102041 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.101951    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert\") pod \"71235c6f-1799-4f13-8eab-61b18c47a390\" (UID: \"71235c6f-1799-4f13-8eab-61b18c47a390\") "
Oct 31 00:26:06.102671 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.102618    4466 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-ca-configmap\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.103906 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.103883    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:26:06.104004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.103986    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:26:06.104326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.104290    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:26:06.104396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.104366    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "kube-api-access-4hmww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:06.104396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.104366    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:26:06.104396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.104388    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token" pod "71235c6f-1799-4f13-8eab-61b18c47a390" (UID: "71235c6f-1799-4f13-8eab-61b18c47a390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:06.204171 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.204134    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.204171 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.204162    4466 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-cacerts\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.204171 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.204172    4466 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-csr-dns-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.204463 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.204183    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmww\" (UniqueName: \"kubernetes.io/projected/71235c6f-1799-4f13-8eab-61b18c47a390-kube-api-access-4hmww\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.204463 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.204193    4466 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71235c6f-1799-4f13-8eab-61b18c47a390-local-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.204463 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.204202    4466 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71235c6f-1799-4f13-8eab-61b18c47a390-istio-kubeconfig\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:06.718903 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.718869    4466 generic.go:356] "Generic (PLEG): container finished" podID="71235c6f-1799-4f13-8eab-61b18c47a390" containerID="d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc" exitCode=0
Oct 31 00:26:06.719102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.718912    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"
Oct 31 00:26:06.719102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.718950    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" event={"ID":"71235c6f-1799-4f13-8eab-61b18c47a390","Type":"ContainerDied","Data":"d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc"}
Oct 31 00:26:06.719102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.719010    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk" event={"ID":"71235c6f-1799-4f13-8eab-61b18c47a390","Type":"ContainerDied","Data":"cdd95c92cd3886c31b9a0b9a577ecfcf78709989513d38831958df2ce9e28ec4"}
Oct 31 00:26:06.719102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.719033    4466 scope.go:122] "RemoveContainer" containerID="d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc"
Oct 31 00:26:06.730141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.730120    4466 scope.go:122] "RemoveContainer" containerID="d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc"
Oct 31 00:26:06.730460 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:26:06.730436    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc\": container with ID starting with d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc not found: ID does not exist" containerID="d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc"
Oct 31 00:26:06.730539 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.730469    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc"} err="failed to get container status \"d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc\": rpc error: code = NotFound desc = could not find container \"d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc\": container with ID starting with d6d5bca56ed5a19700d6ee9a3e15a487eccc125beccfd2b7be23d8ac9af773dc not found: ID does not exist"
Oct 31 00:26:06.893967 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.893922    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"]
Oct 31 00:26:06.946480 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:06.946446    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wqjzk"]
Oct 31 00:26:08.318800 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:26:08.318757    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:26:08.827396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:08.827365    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" path="/var/lib/kubelet/pods/71235c6f-1799-4f13-8eab-61b18c47a390/volumes"
Oct 31 00:26:38.322092 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:26:38.322049    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:26:57.373325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.373281    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"]
Oct 31 00:26:57.373920 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.373530    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy" containerID="cri-o://fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333" gracePeriod=30
Oct 31 00:26:57.425382 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.425341    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"]
Oct 31 00:26:57.425670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.425607    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy" containerID="cri-o://447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd" gracePeriod=30
Oct 31 00:26:57.427880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.427857    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"]
Oct 31 00:26:57.428055 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.428019    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy" containerID="cri-o://27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4" gracePeriod=30
Oct 31 00:26:57.477845 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.477812    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"]
Oct 31 00:26:57.478035 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:57.478015    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" containerID="cri-o://29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06" gracePeriod=30
Oct 31 00:26:59.053232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:59.053194    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-hkgx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:26:59.053682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:26:59.053251    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:27:02.722363 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.722334    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:27:02.767357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.767330    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:27:02.803827 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.803796    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 31 00:27:02.819120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.819092    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 31 00:27:02.881965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.881931    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.881965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.881970    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.881995    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882018    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882046    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882069    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882093    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882124    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882149    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882182    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882215    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.882291 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882250    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882345    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882367    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882407    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882408    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882458    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882489    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882529    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882574    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882579    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882621    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.882884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882847    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.882960    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883005    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883040    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883086    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883142    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883172    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883212    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert\") pod \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\" (UID: \"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883246    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883277    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883310    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883372    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883407    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883437    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.883464 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883461    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883522    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883550    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket\") pod \"ef14768b-ad29-43c8-b2a0-462e0063855a\" (UID: \"ef14768b-ad29-43c8-b2a0-462e0063855a\") "
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883567    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883595    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883624    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token\") pod \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\" (UID: \"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd\") "
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883679    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert\") pod \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\" (UID: \"b56d2a7a-f98b-4fed-b3fa-291df3265d4c\") "
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.883846    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.884205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884117    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.884711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884255    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:02.884711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884660    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884679    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884693    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884706    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884722    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884736    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884750    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884764    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884780    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.884989 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.884792    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.885313 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885288    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.885406 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885382    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885470    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885665    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885670    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885725    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885781    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885800    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885931    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.886006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.885954    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.886559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.886104    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:02.887723 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.887690    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:02.887723 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.887713    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.887883 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.887726    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:02.888235 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888209    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:02.888308 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888263    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "kube-api-access-78csc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.888439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888420    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.888518 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888493    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.888573 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888524    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "kube-api-access-njrrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.888682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888665    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:02.888770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888687    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.888770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888709    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l" pod "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" (UID: "f1d97aa5-d5f6-4263-ac67-5c83b5b742aa"). InnerVolumeSpecName "kube-api-access-k6t7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.888770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888718    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.888770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.888734    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56" pod "ef14768b-ad29-43c8-b2a0-462e0063855a" (UID: "ef14768b-ad29-43c8-b2a0-462e0063855a"). InnerVolumeSpecName "kube-api-access-nhb56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.889310 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.889294    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token" pod "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" (UID: "74bdbd9e-2149-4c46-b48c-5c1ebea3aafd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.890027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.890009    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token" pod "b56d2a7a-f98b-4fed-b3fa-291df3265d4c" (UID: "b56d2a7a-f98b-4fed-b3fa-291df3265d4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.985997    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986023    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986033    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986045    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986054    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986063    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986071    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986080    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986089    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhb56\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-kube-api-access-nhb56\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986101    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6t7l\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-kube-api-access-k6t7l\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986109    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14768b-ad29-43c8-b2a0-462e0063855a-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986118    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986125    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986133    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986143    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986151    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986160    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78csc\" (UniqueName: \"kubernetes.io/projected/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-kube-api-access-78csc\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986169    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986178    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986187    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986195    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986203    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986211    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986219    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986234    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14768b-ad29-43c8-b2a0-462e0063855a-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.986495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:02.986242    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njrrp\" (UniqueName: \"kubernetes.io/projected/b56d2a7a-f98b-4fed-b3fa-291df3265d4c-kube-api-access-njrrp\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:03.039435 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.039402    4466 generic.go:356] "Generic (PLEG): container finished" podID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerID="fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333" exitCode=0
Oct 31 00:27:03.039679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.039464    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"
Oct 31 00:27:03.039679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.039488    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" event={"ID":"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa","Type":"ContainerDied","Data":"fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333"}
Oct 31 00:27:03.039679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.039565    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28" event={"ID":"f1d97aa5-d5f6-4263-ac67-5c83b5b742aa","Type":"ContainerDied","Data":"e61dd5d14a7aeafe5a7555d315391837bf66152c13c1ac0ca5148c38223bae91"}
Oct 31 00:27:03.039679 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.039589    4466 scope.go:122] "RemoveContainer" containerID="fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333"
Oct 31 00:27:03.041351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.041324    4466 generic.go:356] "Generic (PLEG): container finished" podID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerID="27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4" exitCode=0
Oct 31 00:27:03.041482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.041378    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"
Oct 31 00:27:03.041482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.041404    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" event={"ID":"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd","Type":"ContainerDied","Data":"27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4"}
Oct 31 00:27:03.041482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.041456    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv" event={"ID":"74bdbd9e-2149-4c46-b48c-5c1ebea3aafd","Type":"ContainerDied","Data":"1e84eeabfca7c6c82be5955c6d50f39f91adff24f626d1404ef110a02810c3d5"}
Oct 31 00:27:03.043222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.043187    4466 generic.go:356] "Generic (PLEG): container finished" podID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerID="29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06" exitCode=0
Oct 31 00:27:03.043362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.043311    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" event={"ID":"b56d2a7a-f98b-4fed-b3fa-291df3265d4c","Type":"ContainerDied","Data":"29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06"}
Oct 31 00:27:03.043362 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.043353    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7" event={"ID":"b56d2a7a-f98b-4fed-b3fa-291df3265d4c","Type":"ContainerDied","Data":"29bbe6f5e19c763f3bc1c573c44a1f6cbbf04ab694b8178187ef92e2f85406b7"}
Oct 31 00:27:03.043496 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.043422    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.052810    4466 scope.go:122] "RemoveContainer" containerID="fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333"
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.053928    4466 generic.go:356] "Generic (PLEG): container finished" podID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerID="447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd" exitCode=0
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.053993    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" event={"ID":"ef14768b-ad29-43c8-b2a0-462e0063855a","Type":"ContainerDied","Data":"447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd"}
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.054084    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw" event={"ID":"ef14768b-ad29-43c8-b2a0-462e0063855a","Type":"ContainerDied","Data":"b65e4a0064dc09d208f35c4797ac88861fd224177a9da801525878b707e67bde"}
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:03.054085    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333\": container with ID starting with fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333 not found: ID does not exist" containerID="fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333"
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.054119    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333"} err="failed to get container status \"fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333\": rpc error: code = NotFound desc = could not find container \"fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333\": container with ID starting with fefd1d195ad84e4ad0bb94f156a8fb85eeaa33db7ad3753f4caf1b0a0d5d6333 not found: ID does not exist"
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.054143    4466 scope.go:122] "RemoveContainer" containerID="27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4"
Oct 31 00:27:03.054963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.054190    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"
Oct 31 00:27:03.073886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.073863    4466 scope.go:122] "RemoveContainer" containerID="27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4"
Oct 31 00:27:03.074294 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:03.074250    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4\": container with ID starting with 27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4 not found: ID does not exist" containerID="27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4"
Oct 31 00:27:03.074398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.074305    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4"} err="failed to get container status \"27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4\": rpc error: code = NotFound desc = could not find container \"27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4\": container with ID starting with 27559c4c2b67a00b28aea6adcaf6c2078971e2b5a832512bc3130d7d07b1e2d4 not found: ID does not exist"
Oct 31 00:27:03.074398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.074325    4466 scope.go:122] "RemoveContainer" containerID="29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06"
Oct 31 00:27:03.084015 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.083994    4466 scope.go:122] "RemoveContainer" containerID="29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06"
Oct 31 00:27:03.084293 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:03.084272    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06\": container with ID starting with 29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06 not found: ID does not exist" containerID="29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06"
Oct 31 00:27:03.084358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.084303    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06"} err="failed to get container status \"29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06\": rpc error: code = NotFound desc = could not find container \"29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06\": container with ID starting with 29cd1d7a6ce533634d0576af6583a4517b40117e302b22c1a7b54ca1d0c40c06 not found: ID does not exist"
Oct 31 00:27:03.084358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.084320    4466 scope.go:122] "RemoveContainer" containerID="447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd"
Oct 31 00:27:03.093037 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.093015    4466 scope.go:122] "RemoveContainer" containerID="447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd"
Oct 31 00:27:03.093297 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:03.093275    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd\": container with ID starting with 447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd not found: ID does not exist" containerID="447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd"
Oct 31 00:27:03.093355 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.093308    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd"} err="failed to get container status \"447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd\": rpc error: code = NotFound desc = could not find container \"447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd\": container with ID starting with 447a0f94c17d81b8527c59802c80e52dd9c50b8d556046f070f3b0b1bda105cd not found: ID does not exist"
Oct 31 00:27:03.428085 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.428047    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"]
Oct 31 00:27:03.434165 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.434141    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-hkgx7"]
Oct 31 00:27:03.550031 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.549996    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"]
Oct 31 00:27:03.553732 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.553702    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-sgc28"]
Oct 31 00:27:03.672534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.672446    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"]
Oct 31 00:27:03.693347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.693306    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-bd4hv"]
Oct 31 00:27:03.823217 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.823183    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"]
Oct 31 00:27:03.828985 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:03.828957    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-49lgw"]
Oct 31 00:27:04.827253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:04.827218    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" path="/var/lib/kubelet/pods/74bdbd9e-2149-4c46-b48c-5c1ebea3aafd/volumes"
Oct 31 00:27:04.827741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:04.827727    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" path="/var/lib/kubelet/pods/b56d2a7a-f98b-4fed-b3fa-291df3265d4c/volumes"
Oct 31 00:27:04.828144 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:04.828131    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" path="/var/lib/kubelet/pods/ef14768b-ad29-43c8-b2a0-462e0063855a/volumes"
Oct 31 00:27:04.828865 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:04.828851    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" path="/var/lib/kubelet/pods/f1d97aa5-d5f6-4263-ac67-5c83b5b742aa/volumes"
Oct 31 00:27:08.320617 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:08.320579    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:27:13.599796 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.599756    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"]
Oct 31 00:27:13.600240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600197    4466 cpu_manager.go:414] "containerMap: removing container" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy"
Oct 31 00:27:13.600240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600208    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy"
Oct 31 00:27:13.600240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600218    4466 cpu_manager.go:414] "containerMap: removing container" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy"
Oct 31 00:27:13.600240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600224    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy"
Oct 31 00:27:13.600240 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600239    4466 cpu_manager.go:414] "containerMap: removing container" podUID="979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" containerName="revoke-keys"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600244    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" containerName="revoke-keys"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600259    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600264    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600277    4466 cpu_manager.go:414] "containerMap: removing container" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600281    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e83d72a-ac30-48b1-86f1-83b0f29514f7" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600293    4466 cpu_manager.go:414] "containerMap: removing container" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600299    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600309    4466 cpu_manager.go:414] "containerMap: removing container" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" containerName="discovery"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600314    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" containerName="discovery"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600320    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600325    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600383    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="979a3036-14ca-45ce-bdaa-0c6e6b5bfb1e" containerName="revoke-keys"
Oct 31 00:27:13.600389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600393    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56d2a7a-f98b-4fed-b3fa-291df3265d4c" containerName="istio-proxy"
Oct 31 00:27:13.600849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600402    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="71235c6f-1799-4f13-8eab-61b18c47a390" containerName="discovery"
Oct 31 00:27:13.600849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600408    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef14768b-ad29-43c8-b2a0-462e0063855a" containerName="istio-proxy"
Oct 31 00:27:13.600849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600414    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddf13b2-1d84-4d63-89ac-bb8f5ad2f9d5" containerName="istio-proxy"
Oct 31 00:27:13.600849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600422    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d97aa5-d5f6-4263-ac67-5c83b5b742aa" containerName="istio-proxy"
Oct 31 00:27:13.600849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.600428    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bdbd9e-2149-4c46-b48c-5c1ebea3aafd" containerName="istio-proxy"
Oct 31 00:27:13.604925 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.604903    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.658871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.658835    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"]
Oct 31 00:27:13.705327 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.705291    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.705511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.705338    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.705511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.705440    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbb5\" (UniqueName: \"kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.720877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.720841    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:27:13.721049 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.720886    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:27:13.806152 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.806114    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.806405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.806176    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.806405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.806248    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbb5\" (UniqueName: \"kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.806999 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.806973    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.808646 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.808609    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.870237 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.870153    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbb5\" (UniqueName: \"kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5\") pod \"maas-api-revoke-sub-fcb41069-vsz5w\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:13.918937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:13.918893    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:14.288569 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:27:14.288535    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a6711d_61ea_4c59_8f23_5b8e562777f7.slice/crio-0718b08cdca52d0af33ed561ccda4cd870c474c56ed4ca33a67084962c4d5a76 WatchSource:0}: Error finding container 0718b08cdca52d0af33ed561ccda4cd870c474c56ed4ca33a67084962c4d5a76: Status 404 returned error can't find the container with id 0718b08cdca52d0af33ed561ccda4cd870c474c56ed4ca33a67084962c4d5a76
Oct 31 00:27:14.292246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:14.292218    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"]
Oct 31 00:27:15.123458 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:15.123424    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w" event={"ID":"c9a6711d-61ea-4c59-8f23-5b8e562777f7","Type":"ContainerStarted","Data":"d041bb2242c56c4f3fac8ae836c218238935f65b6dedfc299d15cb3e05e16af3"}
Oct 31 00:27:15.123972 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:15.123481    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w" event={"ID":"c9a6711d-61ea-4c59-8f23-5b8e562777f7","Type":"ContainerStarted","Data":"0718b08cdca52d0af33ed561ccda4cd870c474c56ed4ca33a67084962c4d5a76"}
Oct 31 00:27:15.283690 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:15.283604    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w" podStartSLOduration=2.283590266 podStartE2EDuration="2.283590266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:15.283238017 +0000 UTC m=+2740.573786337" watchObservedRunningTime="2026-10-31 00:27:15.283590266 +0000 UTC m=+2740.574138583"
Oct 31 00:27:16.129880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:16.129844    4466 generic.go:356] "Generic (PLEG): container finished" podID="c9a6711d-61ea-4c59-8f23-5b8e562777f7" containerID="d041bb2242c56c4f3fac8ae836c218238935f65b6dedfc299d15cb3e05e16af3" exitCode=0
Oct 31 00:27:16.130310 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:16.129879    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w" event={"ID":"c9a6711d-61ea-4c59-8f23-5b8e562777f7","Type":"ContainerDied","Data":"d041bb2242c56c4f3fac8ae836c218238935f65b6dedfc299d15cb3e05e16af3"}
Oct 31 00:27:17.268772 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.268744    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:17.337517 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.337489    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca\") pod \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") "
Oct 31 00:27:17.337695 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.337571    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5\" (UniqueName: \"kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5\") pod \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") "
Oct 31 00:27:17.337695 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.337616    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca\") pod \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\" (UID: \"c9a6711d-61ea-4c59-8f23-5b8e562777f7\") "
Oct 31 00:27:17.338095 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.338064    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca" pod "c9a6711d-61ea-4c59-8f23-5b8e562777f7" (UID: "c9a6711d-61ea-4c59-8f23-5b8e562777f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:17.339862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.339837    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca" pod "c9a6711d-61ea-4c59-8f23-5b8e562777f7" (UID: "c9a6711d-61ea-4c59-8f23-5b8e562777f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:17.339975 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.339949    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5" pod "c9a6711d-61ea-4c59-8f23-5b8e562777f7" (UID: "c9a6711d-61ea-4c59-8f23-5b8e562777f7"). InnerVolumeSpecName "kube-api-access-mhbb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:17.439381 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.439277    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-xks-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:17.439381 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.439324    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbb5\" (UniqueName: \"kubernetes.io/projected/c9a6711d-61ea-4c59-8f23-5b8e562777f7-kube-api-access-mhbb5\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:17.439381 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:17.439336    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a6711d-61ea-4c59-8f23-5b8e562777f7-maas-api-openshift-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:18.041075 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:18.041044    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"]
Oct 31 00:27:18.059238 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:18.059206    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"]
Oct 31 00:27:18.093356 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:18.093324    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"]
Oct 31 00:27:18.141342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:18.141314    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"
Oct 31 00:27:18.141342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:18.141332    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w" event={"ID":"c9a6711d-61ea-4c59-8f23-5b8e562777f7","Type":"ContainerDied","Data":"0718b08cdca52d0af33ed561ccda4cd870c474c56ed4ca33a67084962c4d5a76"}
Oct 31 00:27:18.141594 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:18.141381    4466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0718b08cdca52d0af33ed561ccda4cd870c474c56ed4ca33a67084962c4d5a76"
Oct 31 00:27:27.600224 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:27.600184    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"]
Oct 31 00:27:27.600700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:27.600472    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy" containerID="cri-o://ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd" gracePeriod=30
Oct 31 00:27:32.859336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.859308    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 31 00:27:32.930485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930395    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930437    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930487    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930519    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930564    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930667    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930726    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.930783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930769    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.931105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930803    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:32.931105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930826    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:32.931105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.930919    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo\") pod \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\" (UID: \"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32\") "
Oct 31 00:27:32.931105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931023    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:32.931105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931042    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:32.931105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931090    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:32.931409 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931386    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:32.931466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931416    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:32.931466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931432    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:32.931466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931447    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:32.931466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.931461    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:32.932953 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.932918    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "kube-api-access-zqcng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:32.933072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.933034    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:32.933222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.933203    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:32.933471 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:32.933449    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy" pod "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" (UID: "d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.032904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.032867    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.032904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.032897    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.032904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.032909    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.033154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.032919    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcng\" (UniqueName: \"kubernetes.io/projected/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32-kube-api-access-zqcng\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.060931 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.060852    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main" containerID="cri-o://2026dfe6ded51d0d2ee6bd741a2007a320ef85c6a74d451a617523f6f96a9799" gracePeriod=45
Oct 31 00:27:33.079263 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.079219    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main" containerID="cri-o://280316e7389022e3323d6ae788a1e2eef6f9224fe5f51688c5d9d9c8da9200d6" gracePeriod=45
Oct 31 00:27:33.119150 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.119105    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" containerID="cri-o://c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a" gracePeriod=45
Oct 31 00:27:33.228607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.228572    4466 generic.go:356] "Generic (PLEG): container finished" podID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerID="2026dfe6ded51d0d2ee6bd741a2007a320ef85c6a74d451a617523f6f96a9799" exitCode=0
Oct 31 00:27:33.228750 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.228667    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" event={"ID":"0594d4aa-17b2-4daa-b15e-1448f224b9c3","Type":"ContainerDied","Data":"2026dfe6ded51d0d2ee6bd741a2007a320ef85c6a74d451a617523f6f96a9799"}
Oct 31 00:27:33.230778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.230755    4466 generic.go:356] "Generic (PLEG): container finished" podID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerID="280316e7389022e3323d6ae788a1e2eef6f9224fe5f51688c5d9d9c8da9200d6" exitCode=0
Oct 31 00:27:33.230892 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.230845    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" event={"ID":"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a","Type":"ContainerDied","Data":"280316e7389022e3323d6ae788a1e2eef6f9224fe5f51688c5d9d9c8da9200d6"}
Oct 31 00:27:33.232886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.232859    4466 generic.go:356] "Generic (PLEG): container finished" podID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerID="ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd" exitCode=0
Oct 31 00:27:33.232998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.232933    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" event={"ID":"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32","Type":"ContainerDied","Data":"ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd"}
Oct 31 00:27:33.232998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.232944    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"
Oct 31 00:27:33.232998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.232983    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt" event={"ID":"d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32","Type":"ContainerDied","Data":"fce92b9bde7b1e059cf0e8fc96203c6113a94668efe345ba28f5369120f6e0a0"}
Oct 31 00:27:33.233158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.233011    4466 scope.go:122] "RemoveContainer" containerID="ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd"
Oct 31 00:27:33.257614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.257532    4466 scope.go:122] "RemoveContainer" containerID="dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"
Oct 31 00:27:33.275661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.275596    4466 scope.go:122] "RemoveContainer" containerID="ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd"
Oct 31 00:27:33.276109 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:33.276080    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd\": container with ID starting with ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd not found: ID does not exist" containerID="ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd"
Oct 31 00:27:33.276235 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.276126    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd"} err="failed to get container status \"ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd\": rpc error: code = NotFound desc = could not find container \"ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd\": container with ID starting with ec9def76299d22f72002d5c8f4e8073381bffe3bd8403ad99575944e45843cbd not found: ID does not exist"
Oct 31 00:27:33.276235 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.276153    4466 scope.go:122] "RemoveContainer" containerID="dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"
Oct 31 00:27:33.276498 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:33.276477    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857\": container with ID starting with dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857 not found: ID does not exist" containerID="dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"
Oct 31 00:27:33.276616 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.276506    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857"} err="failed to get container status \"dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857\": rpc error: code = NotFound desc = could not find container \"dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857\": container with ID starting with dd7beddc73acb3b1a28286168b940dee29e02d04d83dc92a2a231444775dd857 not found: ID does not exist"
Oct 31 00:27:33.294133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.294092    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": dial tcp 10.133.0.46:8000: connect: connection refused"
Oct 31 00:27:33.382331 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.382248    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:27:33.396193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.396161    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 31 00:27:33.407439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.407407    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"]
Oct 31 00:27:33.411978 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.411947    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-8ckgt"]
Oct 31 00:27:33.425708 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.425679    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:27:33.537548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537464    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home\") pod \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") "
Oct 31 00:27:33.537548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537520    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir\") pod \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") "
Oct 31 00:27:33.537548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537541    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs\") pod \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537581    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home\") pod \"55ba300a-9a91-4df8-a822-9e5f21c490c2\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537606    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk\" (UniqueName: \"kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk\") pod \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537644    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm\") pod \"55ba300a-9a91-4df8-a822-9e5f21c490c2\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537733    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache\") pod \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537772    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm\") pod \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537800    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache\") pod \"55ba300a-9a91-4df8-a822-9e5f21c490c2\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") "
Oct 31 00:27:33.537852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537847    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir\") pod \"55ba300a-9a91-4df8-a822-9e5f21c490c2\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") "
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537869    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir" pod "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" (UID: "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537885    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh\" (UniqueName: \"kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh\") pod \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") "
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.537964    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm\") pod \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") "
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.538003    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs\") pod \"55ba300a-9a91-4df8-a822-9e5f21c490c2\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") "
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.538038    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache\") pod \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") "
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.538108    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs\" (UniqueName: \"kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs\") pod \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") "
Oct 31 00:27:33.538196 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.538143    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home" pod "0594d4aa-17b2-4daa-b15e-1448f224b9c3" (UID: "0594d4aa-17b2-4daa-b15e-1448f224b9c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.538648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.538613    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache" pod "55ba300a-9a91-4df8-a822-9e5f21c490c2" (UID: "55ba300a-9a91-4df8-a822-9e5f21c490c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.538866 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.538842    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home" pod "55ba300a-9a91-4df8-a822-9e5f21c490c2" (UID: "55ba300a-9a91-4df8-a822-9e5f21c490c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.539596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.539064    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache" pod "0594d4aa-17b2-4daa-b15e-1448f224b9c3" (UID: "0594d4aa-17b2-4daa-b15e-1448f224b9c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.539596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.539352    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir" pod "55ba300a-9a91-4df8-a822-9e5f21c490c2" (UID: "55ba300a-9a91-4df8-a822-9e5f21c490c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.540964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.540697    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc\" (UniqueName: \"kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc\") pod \"55ba300a-9a91-4df8-a822-9e5f21c490c2\" (UID: \"55ba300a-9a91-4df8-a822-9e5f21c490c2\") "
Oct 31 00:27:33.540964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.540752    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home\") pod \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\" (UID: \"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a\") "
Oct 31 00:27:33.540964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.540794    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir\") pod \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\" (UID: \"0594d4aa-17b2-4daa-b15e-1448f224b9c3\") "
Oct 31 00:27:33.541221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541198    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.541221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541218    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.541221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541227    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.541452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541234    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.541452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541242    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.541452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541250    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.541452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541254    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache" pod "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" (UID: "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.541452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541397    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir" pod "0594d4aa-17b2-4daa-b15e-1448f224b9c3" (UID: "0594d4aa-17b2-4daa-b15e-1448f224b9c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.541760 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541502    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home" pod "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" (UID: "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.541852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541834    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs" pod "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" (UID: "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:33.542002 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.541957    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm" pod "55ba300a-9a91-4df8-a822-9e5f21c490c2" (UID: "55ba300a-9a91-4df8-a822-9e5f21c490c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.542078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.542059    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh" pod "0594d4aa-17b2-4daa-b15e-1448f224b9c3" (UID: "0594d4aa-17b2-4daa-b15e-1448f224b9c3"). InnerVolumeSpecName "kube-api-access-df9dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:33.542298 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.542247    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm" pod "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" (UID: "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.542580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.542558    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs" pod "0594d4aa-17b2-4daa-b15e-1448f224b9c3" (UID: "0594d4aa-17b2-4daa-b15e-1448f224b9c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:33.542933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.542908    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc" pod "55ba300a-9a91-4df8-a822-9e5f21c490c2" (UID: "55ba300a-9a91-4df8-a822-9e5f21c490c2"). InnerVolumeSpecName "kube-api-access-4m6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:33.543359 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.543327    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk" pod "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" (UID: "db8cd2b1-3b6e-429d-9f50-c14bd23cf52a"). InnerVolumeSpecName "kube-api-access-vmjfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:33.543561 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.543545    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm" pod "0594d4aa-17b2-4daa-b15e-1448f224b9c3" (UID: "0594d4aa-17b2-4daa-b15e-1448f224b9c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.544236 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.544214    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs" pod "55ba300a-9a91-4df8-a822-9e5f21c490c2" (UID: "55ba300a-9a91-4df8-a822-9e5f21c490c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:33.642197 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642165    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642197 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642193    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ba300a-9a91-4df8-a822-9e5f21c490c2-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642205    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642215    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642225    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m6jc\" (UniqueName: \"kubernetes.io/projected/55ba300a-9a91-4df8-a822-9e5f21c490c2-kube-api-access-4m6jc\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642242    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642251    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642259    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0594d4aa-17b2-4daa-b15e-1448f224b9c3-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642268    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjfk\" (UniqueName: \"kubernetes.io/projected/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-kube-api-access-vmjfk\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642294    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ba300a-9a91-4df8-a822-9e5f21c490c2-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642303    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.642434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:33.642313    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9dh\" (UniqueName: \"kubernetes.io/projected/0594d4aa-17b2-4daa-b15e-1448f224b9c3-kube-api-access-df9dh\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:34.240090 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.240060    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"
Oct 31 00:27:34.240090 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.240082    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc" event={"ID":"0594d4aa-17b2-4daa-b15e-1448f224b9c3","Type":"ContainerDied","Data":"f4fd34c954b098db5c5c570ff780fb7a11a8eba652f906b4c5228924d02f07c4"}
Oct 31 00:27:34.240681 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.240135    4466 scope.go:122] "RemoveContainer" containerID="2026dfe6ded51d0d2ee6bd741a2007a320ef85c6a74d451a617523f6f96a9799"
Oct 31 00:27:34.241604 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.241577    4466 generic.go:356] "Generic (PLEG): container finished" podID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerID="c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a" exitCode=0
Oct 31 00:27:34.241826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.241619    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"
Oct 31 00:27:34.241826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.241652    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" event={"ID":"55ba300a-9a91-4df8-a822-9e5f21c490c2","Type":"ContainerDied","Data":"c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a"}
Oct 31 00:27:34.241826 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.241700    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs" event={"ID":"55ba300a-9a91-4df8-a822-9e5f21c490c2","Type":"ContainerDied","Data":"bd2918a43503a34dd4777eb669bf9a1ecc920ab1e5f44ac3ef6c2ff1917344eb"}
Oct 31 00:27:34.243309 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.243286    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9" event={"ID":"db8cd2b1-3b6e-429d-9f50-c14bd23cf52a","Type":"ContainerDied","Data":"0cb55e4bf6b35592b92027f3b970af00a5b55a54290c4abaf578f5b242a16e4d"}
Oct 31 00:27:34.243403 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.243315    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"
Oct 31 00:27:34.251046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.251029    4466 scope.go:122] "RemoveContainer" containerID="c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a"
Oct 31 00:27:34.259958 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.259939    4466 scope.go:122] "RemoveContainer" containerID="c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a"
Oct 31 00:27:34.260254 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:34.260231    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a\": container with ID starting with c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a not found: ID does not exist" containerID="c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a"
Oct 31 00:27:34.260316 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.260266    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a"} err="failed to get container status \"c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a\": rpc error: code = NotFound desc = could not find container \"c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a\": container with ID starting with c833094572ae91636ab8a74bd0b9bac1092faa6b2e28d61c089e6bfc73e0652a not found: ID does not exist"
Oct 31 00:27:34.260316 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.260289    4466 scope.go:122] "RemoveContainer" containerID="280316e7389022e3323d6ae788a1e2eef6f9224fe5f51688c5d9d9c8da9200d6"
Oct 31 00:27:34.413067 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.413030    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"]
Oct 31 00:27:34.417141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.417117    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/premium-w4-34f1ce-kserve-6bfc6b564-h27c9"]
Oct 31 00:27:34.531915 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.531829    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"]
Oct 31 00:27:34.537779 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.537741    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/simulator-w4-34f1ce-kserve-568545fcfc-tb7zs"]
Oct 31 00:27:34.650163 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.650120    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"]
Oct 31 00:27:34.656493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.656464    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-2-w4-34f1ce-kserve-b4f856b44-qfhcc"]
Oct 31 00:27:34.826159 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.826076    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" path="/var/lib/kubelet/pods/0594d4aa-17b2-4daa-b15e-1448f224b9c3/volumes"
Oct 31 00:27:34.826622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.826606    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" path="/var/lib/kubelet/pods/55ba300a-9a91-4df8-a822-9e5f21c490c2/volumes"
Oct 31 00:27:34.827052 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.827039    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" path="/var/lib/kubelet/pods/d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32/volumes"
Oct 31 00:27:34.827775 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:34.827760    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" path="/var/lib/kubelet/pods/db8cd2b1-3b6e-429d-9f50-c14bd23cf52a/volumes"
Oct 31 00:27:38.317798 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:38.317759    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:27:42.477614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.477578    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"]
Oct 31 00:27:42.478067 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.477819    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" podUID="aaae1956-e766-462b-b8ae-20fd73f7f48c" containerName="maas-api" containerID="cri-o://d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2" gracePeriod=30
Oct 31 00:27:42.751054 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.751027    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:27:42.939419 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.939386    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\") pod \"aaae1956-e766-462b-b8ae-20fd73f7f48c\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") "
Oct 31 00:27:42.939644 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.939479    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp\" (UniqueName: \"kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp\") pod \"aaae1956-e766-462b-b8ae-20fd73f7f48c\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") "
Oct 31 00:27:42.939644 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.939513    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca\") pod \"aaae1956-e766-462b-b8ae-20fd73f7f48c\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") "
Oct 31 00:27:42.939644 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.939613    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs\") pod \"aaae1956-e766-462b-b8ae-20fd73f7f48c\" (UID: \"aaae1956-e766-462b-b8ae-20fd73f7f48c\") "
Oct 31 00:27:42.939971 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.939940    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca" pod "aaae1956-e766-462b-b8ae-20fd73f7f48c" (UID: "aaae1956-e766-462b-b8ae-20fd73f7f48c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:42.940126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.940056    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae1956-e766-462b-b8ae-20fd73f7f48c-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:42.941624 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.941604    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls" pod "aaae1956-e766-462b-b8ae-20fd73f7f48c" (UID: "aaae1956-e766-462b-b8ae-20fd73f7f48c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:42.941703 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.941682    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp" pod "aaae1956-e766-462b-b8ae-20fd73f7f48c" (UID: "aaae1956-e766-462b-b8ae-20fd73f7f48c"). InnerVolumeSpecName "kube-api-access-692xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:42.941748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:42.941734    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs" pod "aaae1956-e766-462b-b8ae-20fd73f7f48c" (UID: "aaae1956-e766-462b-b8ae-20fd73f7f48c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:43.041682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.041567    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.041682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.041602    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-692xp\" (UniqueName: \"kubernetes.io/projected/aaae1956-e766-462b-b8ae-20fd73f7f48c-kube-api-access-692xp\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.041682 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.041613    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaae1956-e766-462b-b8ae-20fd73f7f48c-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.292170 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.292086    4466 generic.go:356] "Generic (PLEG): container finished" podID="aaae1956-e766-462b-b8ae-20fd73f7f48c" containerID="d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2" exitCode=0
Oct 31 00:27:43.292170 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.292146    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"
Oct 31 00:27:43.292351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.292168    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" event={"ID":"aaae1956-e766-462b-b8ae-20fd73f7f48c","Type":"ContainerDied","Data":"d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2"}
Oct 31 00:27:43.292351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.292226    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7" event={"ID":"aaae1956-e766-462b-b8ae-20fd73f7f48c","Type":"ContainerDied","Data":"6099956ac24489a229e445b09a931044abb1c4b68f8274de8bb2558d90705153"}
Oct 31 00:27:43.292351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.292249    4466 scope.go:122] "RemoveContainer" containerID="d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2"
Oct 31 00:27:43.301452 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.301433    4466 scope.go:122] "RemoveContainer" containerID="d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2"
Oct 31 00:27:43.301755 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:43.301734    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2\": container with ID starting with d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2 not found: ID does not exist" containerID="d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2"
Oct 31 00:27:43.301811 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.301766    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2"} err="failed to get container status \"d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2\": rpc error: code = NotFound desc = could not find container \"d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2\": container with ID starting with d4d14833ce60130429ebd46e48ae15e76c2e73d5fefff84b62d536fab43073c2 not found: ID does not exist"
Oct 31 00:27:43.464596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.464560    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"]
Oct 31 00:27:43.516947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:43.516903    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e5c33-fdc965d8b-f4ff7"]
Oct 31 00:27:44.827046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:44.827016    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaae1956-e766-462b-b8ae-20fd73f7f48c" path="/var/lib/kubelet/pods/aaae1956-e766-462b-b8ae-20fd73f7f48c/volumes"
Oct 31 00:27:48.569699 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.569657    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"]
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570093    4466 cpu_manager.go:414] "containerMap: removing container" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570105    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570116    4466 cpu_manager.go:414] "containerMap: removing container" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570121    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570129    4466 cpu_manager.go:414] "containerMap: removing container" podUID="c9a6711d-61ea-4c59-8f23-5b8e562777f7" containerName="revoke-keys"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570134    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a6711d-61ea-4c59-8f23-5b8e562777f7" containerName="revoke-keys"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570147    4466 cpu_manager.go:414] "containerMap: removing container" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570157    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570169    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570174    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570188    4466 cpu_manager.go:414] "containerMap: removing container" podUID="aaae1956-e766-462b-b8ae-20fd73f7f48c" containerName="maas-api"
Oct 31 00:27:48.570192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570192    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaae1956-e766-462b-b8ae-20fd73f7f48c" containerName="maas-api"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570202    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570207    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570278    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570310    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaae1956-e766-462b-b8ae-20fd73f7f48c" containerName="maas-api"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570320    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90e56b0-a1e4-42ad-bd0f-e26a3cd96e32" containerName="istio-proxy"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570329    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="0594d4aa-17b2-4daa-b15e-1448f224b9c3" containerName="main"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570334    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ba300a-9a91-4df8-a822-9e5f21c490c2" containerName="main"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570341    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a6711d-61ea-4c59-8f23-5b8e562777f7" containerName="revoke-keys"
Oct 31 00:27:48.570608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.570347    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8cd2b1-3b6e-429d-9f50-c14bd23cf52a" containerName="main"
Oct 31 00:27:48.574451 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.574429    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.601804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.601772    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.601994 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.601833    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbjd\" (UniqueName: \"kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.601994 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.601949    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.630548 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.630503    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"]
Oct 31 00:27:48.687782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.687743    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:27:48.687782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.687766    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:27:48.703040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.703003    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.703223 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.703072    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbjd\" (UniqueName: \"kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.703223 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.703151    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.703817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.703797    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.705513 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.705486    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.768397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.768357    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbjd\" (UniqueName: \"kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd\") pod \"maas-api-revoke-sub-30504b90-sw74j\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:48.891027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:48.890993    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:49.268767 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:27:49.268737    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137f6a70_2208_4f31_9371_d59d7f02c91d.slice/crio-5bf173fb98630e5d1bdd3730edb7db14f688c9eb7cdea75ecb55a1299a0cbe34 WatchSource:0}: Error finding container 5bf173fb98630e5d1bdd3730edb7db14f688c9eb7cdea75ecb55a1299a0cbe34: Status 404 returned error can't find the container with id 5bf173fb98630e5d1bdd3730edb7db14f688c9eb7cdea75ecb55a1299a0cbe34
Oct 31 00:27:49.270181 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:49.270154    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"]
Oct 31 00:27:49.323038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:49.323003    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j" event={"ID":"137f6a70-2208-4f31-9371-d59d7f02c91d","Type":"ContainerStarted","Data":"5bf173fb98630e5d1bdd3730edb7db14f688c9eb7cdea75ecb55a1299a0cbe34"}
Oct 31 00:27:50.330373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:50.330334    4466 generic.go:356] "Generic (PLEG): container finished" podID="137f6a70-2208-4f31-9371-d59d7f02c91d" containerID="dd68b8ae9d7951a32e0e13696bca7e8005c272728ff3e46463c1fbc8f9475a36" exitCode=0
Oct 31 00:27:50.330802 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:50.330418    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j" event={"ID":"137f6a70-2208-4f31-9371-d59d7f02c91d","Type":"ContainerDied","Data":"dd68b8ae9d7951a32e0e13696bca7e8005c272728ff3e46463c1fbc8f9475a36"}
Oct 31 00:27:51.475848 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.475820    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:51.532856 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.532811    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca\") pod \"137f6a70-2208-4f31-9371-d59d7f02c91d\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") "
Oct 31 00:27:51.533072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.532885    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca\") pod \"137f6a70-2208-4f31-9371-d59d7f02c91d\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") "
Oct 31 00:27:51.533072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.532958    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd\" (UniqueName: \"kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd\") pod \"137f6a70-2208-4f31-9371-d59d7f02c91d\" (UID: \"137f6a70-2208-4f31-9371-d59d7f02c91d\") "
Oct 31 00:27:51.533332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.533303    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca" pod "137f6a70-2208-4f31-9371-d59d7f02c91d" (UID: "137f6a70-2208-4f31-9371-d59d7f02c91d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:51.535107 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.535065    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca" pod "137f6a70-2208-4f31-9371-d59d7f02c91d" (UID: "137f6a70-2208-4f31-9371-d59d7f02c91d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:51.535222 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.535105    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd" pod "137f6a70-2208-4f31-9371-d59d7f02c91d" (UID: "137f6a70-2208-4f31-9371-d59d7f02c91d"). InnerVolumeSpecName "kube-api-access-nsbjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:51.634188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.634150    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-xks-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:51.634188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.634185    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137f6a70-2208-4f31-9371-d59d7f02c91d-maas-api-openshift-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:51.634188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.634198    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbjd\" (UniqueName: \"kubernetes.io/projected/137f6a70-2208-4f31-9371-d59d7f02c91d-kube-api-access-nsbjd\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:51.705998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.705960    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"]
Oct 31 00:27:51.706277 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:51.706245    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" containerID="cri-o://afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081" gracePeriod=30
Oct 31 00:27:52.341484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:52.341448    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j" event={"ID":"137f6a70-2208-4f31-9371-d59d7f02c91d","Type":"ContainerDied","Data":"5bf173fb98630e5d1bdd3730edb7db14f688c9eb7cdea75ecb55a1299a0cbe34"}
Oct 31 00:27:52.341484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:52.341480    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"
Oct 31 00:27:52.341484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:52.341501    4466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf173fb98630e5d1bdd3730edb7db14f688c9eb7cdea75ecb55a1299a0cbe34"
Oct 31 00:27:53.867036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:53.866994    4466 patch_prober.go:29] interesting pod/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:27:53.867441 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:53.867066    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:27:56.964795 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:56.964760    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:27:57.090801 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.090772    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.090979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.090888    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.090979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.090924    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.090979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.090953    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.091146 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.090993    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.091146 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091026    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.091146 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091061    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.091146 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091129    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.091358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091181    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo\") pod \"e1eed02a-eaf2-452f-8b00-a234d021547a\" (UID: \"e1eed02a-eaf2-452f-8b00-a234d021547a\") "
Oct 31 00:27:57.091358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091251    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:57.091358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091334    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:57.091525 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091358    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:57.091525 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091492    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:57.091783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091751    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.091783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091776    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1eed02a-eaf2-452f-8b00-a234d021547a-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.092021 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091792    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.092021 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091840    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.092021 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.091860    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:57.093216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.093191    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:57.093216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.093207    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "kube-api-access-jj29p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:57.093367 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.093282    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:57.093406 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.093380    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo" pod "e1eed02a-eaf2-452f-8b00-a234d021547a" (UID: "e1eed02a-eaf2-452f-8b00-a234d021547a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:57.193357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.193309    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.193357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.193347    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1eed02a-eaf2-452f-8b00-a234d021547a-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.193357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.193357    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.193357 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.193366    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1eed02a-eaf2-452f-8b00-a234d021547a-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.193733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.193376    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj29p\" (UniqueName: \"kubernetes.io/projected/e1eed02a-eaf2-452f-8b00-a234d021547a-kube-api-access-jj29p\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.208312 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.208277    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"]
Oct 31 00:27:57.233960 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.233918    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"]
Oct 31 00:27:57.376443 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.376357    4466 generic.go:356] "Generic (PLEG): container finished" podID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerID="afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081" exitCode=0
Oct 31 00:27:57.376443 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.376399    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"
Oct 31 00:27:57.376672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.376467    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" event={"ID":"e1eed02a-eaf2-452f-8b00-a234d021547a","Type":"ContainerDied","Data":"afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081"}
Oct 31 00:27:57.376672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.376530    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w" event={"ID":"e1eed02a-eaf2-452f-8b00-a234d021547a","Type":"ContainerDied","Data":"6fa22426ce10a7cfb6ae0c594c8cf11c1a114500952526d5566896cf21aa48b4"}
Oct 31 00:27:57.376672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.376562    4466 scope.go:122] "RemoveContainer" containerID="afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081"
Oct 31 00:27:57.386786 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.386769    4466 scope.go:122] "RemoveContainer" containerID="8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"
Oct 31 00:27:57.395800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.395777    4466 scope.go:122] "RemoveContainer" containerID="afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081"
Oct 31 00:27:57.396072 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:57.396052    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081\": container with ID starting with afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081 not found: ID does not exist" containerID="afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081"
Oct 31 00:27:57.396118 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.396084    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081"} err="failed to get container status \"afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081\": rpc error: code = NotFound desc = could not find container \"afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081\": container with ID starting with afb9ab878e514c157ce7027f0e51d1b7cf21b925d3b635eb30625808738d0081 not found: ID does not exist"
Oct 31 00:27:57.396118 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.396102    4466 scope.go:122] "RemoveContainer" containerID="8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"
Oct 31 00:27:57.396351 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:27:57.396333    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455\": container with ID starting with 8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455 not found: ID does not exist" containerID="8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"
Oct 31 00:27:57.396415 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.396361    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455"} err="failed to get container status \"8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455\": rpc error: code = NotFound desc = could not find container \"8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455\": container with ID starting with 8b0d42810814fda941cc75b3e8ceeee9d642f0bcaa881232ebe4a59e488ab455 not found: ID does not exist"
Oct 31 00:27:57.573341 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.573301    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"]
Oct 31 00:27:57.577583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:57.577556    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1e5c33-openshift-default-6f5f6b8dc9-lmv7w"]
Oct 31 00:27:58.826539 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:27:58.826504    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" path="/var/lib/kubelet/pods/e1eed02a-eaf2-452f-8b00-a234d021547a/volumes"
Oct 31 00:28:08.316865 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:28:08.316818    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:28:12.235829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.235791    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main" containerID="cri-o://42c9937523f02d37d3c548222d4daa70796d7ebc3fd50ab74021069ce7047113" gracePeriod=45
Oct 31 00:28:12.253363 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.253332    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" containerID="cri-o://81112adbe3445e729dbfb7aedb8e53371b819de9a93bd4866a8855bd3a74624f" gracePeriod=45
Oct 31 00:28:12.256748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.256719    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 31 00:28:12.453709 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.453626    4466 generic.go:356] "Generic (PLEG): container finished" podID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerID="42c9937523f02d37d3c548222d4daa70796d7ebc3fd50ab74021069ce7047113" exitCode=0
Oct 31 00:28:12.453925 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.453804    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" event={"ID":"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f","Type":"ContainerDied","Data":"42c9937523f02d37d3c548222d4daa70796d7ebc3fd50ab74021069ce7047113"}
Oct 31 00:28:12.456877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.456847    4466 generic.go:356] "Generic (PLEG): container finished" podID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerID="81112adbe3445e729dbfb7aedb8e53371b819de9a93bd4866a8855bd3a74624f" exitCode=0
Oct 31 00:28:12.457036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.456987    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" event={"ID":"9c41c89a-9d80-455f-9821-fa98584a88aa","Type":"ContainerDied","Data":"81112adbe3445e729dbfb7aedb8e53371b819de9a93bd4866a8855bd3a74624f"}
Oct 31 00:28:12.508969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.508944    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:28:12.537675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.537651    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:28:12.660472 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660435    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf\" (UniqueName: \"kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf\") pod \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") "
Oct 31 00:28:12.660670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660514    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs\") pod \"9c41c89a-9d80-455f-9821-fa98584a88aa\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") "
Oct 31 00:28:12.660670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660569    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache\") pod \"9c41c89a-9d80-455f-9821-fa98584a88aa\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") "
Oct 31 00:28:12.660670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660600    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm\") pod \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") "
Oct 31 00:28:12.660878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660693    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm\") pod \"9c41c89a-9d80-455f-9821-fa98584a88aa\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") "
Oct 31 00:28:12.660878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660791    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g\" (UniqueName: \"kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g\") pod \"9c41c89a-9d80-455f-9821-fa98584a88aa\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") "
Oct 31 00:28:12.660878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660819    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home\") pod \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") "
Oct 31 00:28:12.660878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660856    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir\") pod \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") "
Oct 31 00:28:12.661088 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660890    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir\") pod \"9c41c89a-9d80-455f-9821-fa98584a88aa\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") "
Oct 31 00:28:12.661088 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660932    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs\") pod \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") "
Oct 31 00:28:12.661088 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.660977    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home\") pod \"9c41c89a-9d80-455f-9821-fa98584a88aa\" (UID: \"9c41c89a-9d80-455f-9821-fa98584a88aa\") "
Oct 31 00:28:12.661088 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.661011    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache\") pod \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\" (UID: \"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f\") "
Oct 31 00:28:12.662158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.661688    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir" pod "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" (UID: "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.662158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.661836    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache" pod "9c41c89a-9d80-455f-9821-fa98584a88aa" (UID: "9c41c89a-9d80-455f-9821-fa98584a88aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.662158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.661893    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache" pod "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" (UID: "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.662158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.661999    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home" pod "9c41c89a-9d80-455f-9821-fa98584a88aa" (UID: "9c41c89a-9d80-455f-9821-fa98584a88aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.662158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.662016    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home" pod "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" (UID: "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.662158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.662120    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir" pod "9c41c89a-9d80-455f-9821-fa98584a88aa" (UID: "9c41c89a-9d80-455f-9821-fa98584a88aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.663622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.663595    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs" pod "9c41c89a-9d80-455f-9821-fa98584a88aa" (UID: "9c41c89a-9d80-455f-9821-fa98584a88aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:12.664098 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.664073    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm" pod "9c41c89a-9d80-455f-9821-fa98584a88aa" (UID: "9c41c89a-9d80-455f-9821-fa98584a88aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.664665 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.664617    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm" pod "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" (UID: "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.664768 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.664739    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf" pod "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" (UID: "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f"). InnerVolumeSpecName "kube-api-access-69qbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:12.664898 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.664879    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g" pod "9c41c89a-9d80-455f-9821-fa98584a88aa" (UID: "9c41c89a-9d80-455f-9821-fa98584a88aa"). InnerVolumeSpecName "kube-api-access-pml9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:12.665094 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.665073    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs" pod "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" (UID: "36e56b66-19c9-45b6-b6eb-f12ed36f5b6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:12.762934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762864    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qbf\" (UniqueName: \"kubernetes.io/projected/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-kube-api-access-69qbf\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.762934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762892    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41c89a-9d80-455f-9821-fa98584a88aa-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.762934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762905    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.762934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762914    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.762934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762925    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.762934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762934    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pml9g\" (UniqueName: \"kubernetes.io/projected/9c41c89a-9d80-455f-9821-fa98584a88aa-kube-api-access-pml9g\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.763281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762943    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.763281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762951    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.763281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762959    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.763281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762966    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.763281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762973    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41c89a-9d80-455f-9821-fa98584a88aa-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.763281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:12.762982    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.463344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.463312    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"
Oct 31 00:28:13.463884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.463347    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb" event={"ID":"36e56b66-19c9-45b6-b6eb-f12ed36f5b6f","Type":"ContainerDied","Data":"334181cd96565c11ec9e003aa7300698a6051075007f881d2303537b9435dcdb"}
Oct 31 00:28:13.463884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.463408    4466 scope.go:122] "RemoveContainer" containerID="42c9937523f02d37d3c548222d4daa70796d7ebc3fd50ab74021069ce7047113"
Oct 31 00:28:13.464901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.464876    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl" event={"ID":"9c41c89a-9d80-455f-9821-fa98584a88aa","Type":"ContainerDied","Data":"6d101ad1aed75f7585d8e9b82722239e5ce5d11a7b89239ff6d4d04defd572b1"}
Oct 31 00:28:13.465002 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.464929    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"
Oct 31 00:28:13.475703 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.475685    4466 scope.go:122] "RemoveContainer" containerID="81112adbe3445e729dbfb7aedb8e53371b819de9a93bd4866a8855bd3a74624f"
Oct 31 00:28:13.634577 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.634541    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"]
Oct 31 00:28:13.682393 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.682362    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-w3-986d37-kserve-646749ff-wh9mb"]
Oct 31 00:28:13.802260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.802218    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"]
Oct 31 00:28:13.804057 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:13.804038    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/simulator-w3-986d37-kserve-5756b8b7d9-h59dl"]
Oct 31 00:28:14.825736 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:14.825700    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" path="/var/lib/kubelet/pods/36e56b66-19c9-45b6-b6eb-f12ed36f5b6f/volumes"
Oct 31 00:28:14.826175 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:14.826160    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" path="/var/lib/kubelet/pods/9c41c89a-9d80-455f-9821-fa98584a88aa/volumes"
Oct 31 00:28:17.487865 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.487833    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"]
Oct 31 00:28:17.488412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488390    4466 cpu_manager.go:414] "containerMap: removing container" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main"
Oct 31 00:28:17.488511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488416    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main"
Oct 31 00:28:17.488511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488435    4466 cpu_manager.go:414] "containerMap: removing container" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main"
Oct 31 00:28:17.488511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488444    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main"
Oct 31 00:28:17.488511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488472    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy"
Oct 31 00:28:17.488511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488480    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488514    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488521    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488537    4466 cpu_manager.go:414] "containerMap: removing container" podUID="137f6a70-2208-4f31-9371-d59d7f02c91d" containerName="revoke-keys"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488545    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137f6a70-2208-4f31-9371-d59d7f02c91d" containerName="revoke-keys"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488656    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488674    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eed02a-eaf2-452f-8b00-a234d021547a" containerName="istio-proxy"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488686    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="137f6a70-2208-4f31-9371-d59d7f02c91d" containerName="revoke-keys"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488699    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c41c89a-9d80-455f-9821-fa98584a88aa" containerName="main"
Oct 31 00:28:17.488800 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.488708    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e56b66-19c9-45b6-b6eb-f12ed36f5b6f" containerName="main"
Oct 31 00:28:17.493401 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.493372    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.548934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.548892    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"]
Oct 31 00:28:17.582386 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.582358    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:28:17.598396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.598371    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:28:17.611859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.611833    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfng\" (UniqueName: \"kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.612001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.611967    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.612047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.612006    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.713304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.713265    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.713562 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.713322    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.713562 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.713449    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfng\" (UniqueName: \"kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.714149 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.714127    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.715936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.715912    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.782058 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.781996    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfng\" (UniqueName: \"kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng\") pod \"maas-api-revoke-sub-ec0166c0-vs2lx\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:17.805992 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:17.805963    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:18.172326 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:28:18.172295    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d726f2_fb3c_40e8_9386_cd2c3493f882.slice/crio-018825f522551669df8bfef45c928fae7f8a87929f3a39c9ebf0387d8b42d01b WatchSource:0}: Error finding container 018825f522551669df8bfef45c928fae7f8a87929f3a39c9ebf0387d8b42d01b: Status 404 returned error can't find the container with id 018825f522551669df8bfef45c928fae7f8a87929f3a39c9ebf0387d8b42d01b
Oct 31 00:28:18.175708 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:18.175686    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"]
Oct 31 00:28:18.492457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:18.492419    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx" event={"ID":"b1d726f2-fb3c-40e8-9386-cd2c3493f882","Type":"ContainerStarted","Data":"29574877a6e2bb239d91c794fa834f269b120794a2f69f5b7d30c3a9b6464638"}
Oct 31 00:28:18.492896 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:18.492476    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx" event={"ID":"b1d726f2-fb3c-40e8-9386-cd2c3493f882","Type":"ContainerStarted","Data":"018825f522551669df8bfef45c928fae7f8a87929f3a39c9ebf0387d8b42d01b"}
Oct 31 00:28:18.613184 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:18.613130    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx" podStartSLOduration=1.613116966 podStartE2EDuration="1.613116966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:18.612663047 +0000 UTC m=+2803.903211364" watchObservedRunningTime="2026-10-31 00:28:18.613116966 +0000 UTC m=+2803.903665325"
Oct 31 00:28:19.499565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:19.499515    4466 generic.go:356] "Generic (PLEG): container finished" podID="b1d726f2-fb3c-40e8-9386-cd2c3493f882" containerID="29574877a6e2bb239d91c794fa834f269b120794a2f69f5b7d30c3a9b6464638" exitCode=0
Oct 31 00:28:19.500016 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:19.499601    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx" event={"ID":"b1d726f2-fb3c-40e8-9386-cd2c3493f882","Type":"ContainerDied","Data":"29574877a6e2bb239d91c794fa834f269b120794a2f69f5b7d30c3a9b6464638"}
Oct 31 00:28:20.643253 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.643226    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:20.743899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.743868    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca\") pod \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") "
Oct 31 00:28:20.744092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.743915    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca\") pod \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") "
Oct 31 00:28:20.744092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.743936    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng\" (UniqueName: \"kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng\") pod \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\" (UID: \"b1d726f2-fb3c-40e8-9386-cd2c3493f882\") "
Oct 31 00:28:20.744250 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.744226    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca" pod "b1d726f2-fb3c-40e8-9386-cd2c3493f882" (UID: "b1d726f2-fb3c-40e8-9386-cd2c3493f882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:20.746012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.745977    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca" pod "b1d726f2-fb3c-40e8-9386-cd2c3493f882" (UID: "b1d726f2-fb3c-40e8-9386-cd2c3493f882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:20.746129 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.746070    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng" pod "b1d726f2-fb3c-40e8-9386-cd2c3493f882" (UID: "b1d726f2-fb3c-40e8-9386-cd2c3493f882"). InnerVolumeSpecName "kube-api-access-ntfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:20.845348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.845315    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-openshift-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:20.845348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.845343    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d726f2-fb3c-40e8-9386-cd2c3493f882-maas-api-xks-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:20.845348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:20.845354    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfng\" (UniqueName: \"kubernetes.io/projected/b1d726f2-fb3c-40e8-9386-cd2c3493f882-kube-api-access-ntfng\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:21.510984 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:21.510955    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"
Oct 31 00:28:21.510984 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:21.510971    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx" event={"ID":"b1d726f2-fb3c-40e8-9386-cd2c3493f882","Type":"ContainerDied","Data":"018825f522551669df8bfef45c928fae7f8a87929f3a39c9ebf0387d8b42d01b"}
Oct 31 00:28:21.511192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:21.511020    4466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="018825f522551669df8bfef45c928fae7f8a87929f3a39c9ebf0387d8b42d01b"
Oct 31 00:28:33.306067 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.306031    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"]
Oct 31 00:28:33.306493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.306249    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn" podUID="6c0091c1-6953-4cee-8eb7-54926c156ea5" containerName="maas-api" containerID="cri-o://cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29" gracePeriod=30
Oct 31 00:28:33.558901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.558837    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 31 00:28:33.576651 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.576605    4466 generic.go:356] "Generic (PLEG): container finished" podID="6c0091c1-6953-4cee-8eb7-54926c156ea5" containerID="cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29" exitCode=0
Oct 31 00:28:33.576833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.576682    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"
Oct 31 00:28:33.576833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.576689    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn" event={"ID":"6c0091c1-6953-4cee-8eb7-54926c156ea5","Type":"ContainerDied","Data":"cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29"}
Oct 31 00:28:33.576833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.576750    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn" event={"ID":"6c0091c1-6953-4cee-8eb7-54926c156ea5","Type":"ContainerDied","Data":"fbe69d0623bbe46ea06d3d4561cbd82f6d19a616395119b96ff77bc7285bbc5a"}
Oct 31 00:28:33.576833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.576777    4466 scope.go:122] "RemoveContainer" containerID="cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29"
Oct 31 00:28:33.585474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.585449    4466 scope.go:122] "RemoveContainer" containerID="cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29"
Oct 31 00:28:33.585748 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:28:33.585730    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29\": container with ID starting with cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29 not found: ID does not exist" containerID="cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29"
Oct 31 00:28:33.585804 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.585761    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29"} err="failed to get container status \"cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29\": rpc error: code = NotFound desc = could not find container \"cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29\": container with ID starting with cf7c4ba433f149cf74e9c7236f3005fb16ae0b40089cd52654abbd1180952c29 not found: ID does not exist"
Oct 31 00:28:33.678426 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.678390    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq\" (UniqueName: \"kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq\") pod \"6c0091c1-6953-4cee-8eb7-54926c156ea5\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") "
Oct 31 00:28:33.678654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.678513    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca\") pod \"6c0091c1-6953-4cee-8eb7-54926c156ea5\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") "
Oct 31 00:28:33.678654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.678574    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs\") pod \"6c0091c1-6953-4cee-8eb7-54926c156ea5\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") "
Oct 31 00:28:33.678785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.678712    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls\") pod \"6c0091c1-6953-4cee-8eb7-54926c156ea5\" (UID: \"6c0091c1-6953-4cee-8eb7-54926c156ea5\") "
Oct 31 00:28:33.679040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.678867    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca" pod "6c0091c1-6953-4cee-8eb7-54926c156ea5" (UID: "6c0091c1-6953-4cee-8eb7-54926c156ea5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:33.679040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.679015    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0091c1-6953-4cee-8eb7-54926c156ea5-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:33.681164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.681117    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls" pod "6c0091c1-6953-4cee-8eb7-54926c156ea5" (UID: "6c0091c1-6953-4cee-8eb7-54926c156ea5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:33.681842 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.681776    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq" pod "6c0091c1-6953-4cee-8eb7-54926c156ea5" (UID: "6c0091c1-6953-4cee-8eb7-54926c156ea5"). InnerVolumeSpecName "kube-api-access-drgfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:33.681942 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.681833    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs" pod "6c0091c1-6953-4cee-8eb7-54926c156ea5" (UID: "6c0091c1-6953-4cee-8eb7-54926c156ea5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:33.779964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.779927    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:33.779964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.779962    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c0091c1-6953-4cee-8eb7-54926c156ea5-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:33.780233 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:33.779976    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgfq\" (UniqueName: \"kubernetes.io/projected/6c0091c1-6953-4cee-8eb7-54926c156ea5-kube-api-access-drgfq\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:34.007851 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:34.007808    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"]
Oct 31 00:28:34.013793 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:34.013764    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-34f1ce-7d8dd6b579-5ggzn"]
Oct 31 00:28:34.832691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:34.832658    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0091c1-6953-4cee-8eb7-54926c156ea5" path="/var/lib/kubelet/pods/6c0091c1-6953-4cee-8eb7-54926c156ea5/volumes"
Oct 31 00:28:38.318452 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:28:38.318415    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:28:42.863078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:42.863043    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"]
Oct 31 00:28:42.863560 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:42.863278    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy" containerID="cri-o://6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c" gracePeriod=30
Oct 31 00:28:48.115103 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.115074    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:28:48.226473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226386    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226473 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226444    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226496    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226530    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226554    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226580    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226605    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226657    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.226727 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226719    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy\") pod \"e6a87bf5-30da-4901-8be0-e399f44cf476\" (UID: \"e6a87bf5-30da-4901-8be0-e399f44cf476\") "
Oct 31 00:28:48.227101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226837    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.227101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.226851    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.227101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.227064    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.227272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.227092    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.227272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.227133    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.227272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.227154    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.227393 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.227267    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:48.228832 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.228809    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:28:48.228963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.228842    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "kube-api-access-bbkrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:48.229092 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.229075    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:48.229164 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.229129    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy" pod "e6a87bf5-30da-4901-8be0-e399f44cf476" (UID: "e6a87bf5-30da-4901-8be0-e399f44cf476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.327684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327648    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a87bf5-30da-4901-8be0-e399f44cf476-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.327684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327679    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbkrk\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-kube-api-access-bbkrk\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.327684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327689    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.327914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327699    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.327914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327707    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.327914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327716    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.327914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.327724    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a87bf5-30da-4901-8be0-e399f44cf476-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.649188 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.649155    4466 generic.go:356] "Generic (PLEG): container finished" podID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerID="6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c" exitCode=0
Oct 31 00:28:48.649344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.649198    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"
Oct 31 00:28:48.649344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.649232    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" event={"ID":"e6a87bf5-30da-4901-8be0-e399f44cf476","Type":"ContainerDied","Data":"6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c"}
Oct 31 00:28:48.649344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.649311    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt" event={"ID":"e6a87bf5-30da-4901-8be0-e399f44cf476","Type":"ContainerDied","Data":"c13d72d86845f87d0f731ef1cdb869805fce142fc577c0cd7eac5b4afa157653"}
Oct 31 00:28:48.649344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.649336    4466 scope.go:122] "RemoveContainer" containerID="6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c"
Oct 31 00:28:48.658438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.658420    4466 scope.go:122] "RemoveContainer" containerID="6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c"
Oct 31 00:28:48.658766 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:28:48.658744    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c\": container with ID starting with 6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c not found: ID does not exist" containerID="6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c"
Oct 31 00:28:48.658876 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.658773    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c"} err="failed to get container status \"6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c\": rpc error: code = NotFound desc = could not find container \"6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c\": container with ID starting with 6f54d39d51621051b917ab1c1815d11042662bf91f3ee9f21d223cd57263d14c not found: ID does not exist"
Oct 31 00:28:48.825714 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.825687    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"]
Oct 31 00:28:48.875703 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:48.875675    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-vx2nt"]
Oct 31 00:28:50.825779 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:28:50.825748    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" path="/var/lib/kubelet/pods/e6a87bf5-30da-4901-8be0-e399f44cf476/volumes"
Oct 31 00:29:07.642375 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:07.642341    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"]
Oct 31 00:29:07.642982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:07.642585    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy" containerID="cri-o://6e720e5c2a12303530d8539a230519761a94a89992d3a7e3ce13c030ae5a4ce8" gracePeriod=30
Oct 31 00:29:08.317243 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:29:08.317205    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:29:12.765919 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.765889    4466 generic.go:356] "Generic (PLEG): container finished" podID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerID="6e720e5c2a12303530d8539a230519761a94a89992d3a7e3ce13c030ae5a4ce8" exitCode=0
Oct 31 00:29:12.766258 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.765956    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" event={"ID":"e0be6323-9b09-4367-a147-c5e83b1d4600","Type":"ContainerDied","Data":"6e720e5c2a12303530d8539a230519761a94a89992d3a7e3ce13c030ae5a4ce8"}
Oct 31 00:29:12.900697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.900662    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 31 00:29:12.975207 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975114    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975207 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975163    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975207 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975187    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975225    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975267    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975297    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975333    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975390    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975427    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975442    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert\") pod \"e0be6323-9b09-4367-a147-c5e83b1d4600\" (UID: \"e0be6323-9b09-4367-a147-c5e83b1d4600\") "
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975558    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:12.975585 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975581    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975691    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975797    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.975985    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0be6323-9b09-4367-a147-c5e83b1d4600-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.976010    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.976029    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.976042    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.976112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.976056    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:12.977856 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.977828    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:12.977967 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.977952    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:12.978014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.977965    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:12.978014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:12.977950    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc" pod "e0be6323-9b09-4367-a147-c5e83b1d4600" (UID: "e0be6323-9b09-4367-a147-c5e83b1d4600"). InnerVolumeSpecName "kube-api-access-4hssc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:13.077104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.077069    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:13.077104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.077098    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:13.077104 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.077108    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hssc\" (UniqueName: \"kubernetes.io/projected/e0be6323-9b09-4367-a147-c5e83b1d4600-kube-api-access-4hssc\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:13.077343 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.077118    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0be6323-9b09-4367-a147-c5e83b1d4600-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:13.312895 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.312800    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"]
Oct 31 00:29:13.313102 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.313078    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy" containerID="cri-o://f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978" gracePeriod=30
Oct 31 00:29:13.772475 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.772441    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g" event={"ID":"e0be6323-9b09-4367-a147-c5e83b1d4600","Type":"ContainerDied","Data":"6d32bce3bec6925d1ed61ce4f53b6c8a0248cf06523de6da2a5c58dcb717b897"}
Oct 31 00:29:13.772475 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.772461    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"
Oct 31 00:29:13.773003 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.772504    4466 scope.go:122] "RemoveContainer" containerID="6e720e5c2a12303530d8539a230519761a94a89992d3a7e3ce13c030ae5a4ce8"
Oct 31 00:29:13.945245 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.945213    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"]
Oct 31 00:29:13.997387 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:13.997352    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-hrh5g"]
Oct 31 00:29:14.825423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:14.825388    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" path="/var/lib/kubelet/pods/e0be6323-9b09-4367-a147-c5e83b1d4600/volumes"
Oct 31 00:29:18.561793 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.561767    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:29:18.739192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739148    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739206    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739239    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739314    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739349    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739390    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739439    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739495    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:18.739680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739502    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739680 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739587    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token\") pod \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\" (UID: \"e5e949a8-6bb3-4108-818f-dc99b419fdc0\") "
Oct 31 00:29:18.739891 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739826    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:18.739891 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.739827    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:18.740025 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.740001    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:18.740141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.740118    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.740141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.740139    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.740329 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.740153    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.740329 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.740170    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.740566 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.740548    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:18.742337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.742309    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:29:18.742493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.742469    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "kube-api-access-cclf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:18.742582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.742528    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:18.743197 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.743177    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy" pod "e5e949a8-6bb3-4108-818f-dc99b419fdc0" (UID: "e5e949a8-6bb3-4108-818f-dc99b419fdc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:18.799928 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.799892    4466 generic.go:356] "Generic (PLEG): container finished" podID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerID="f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978" exitCode=0
Oct 31 00:29:18.800122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.799949    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"
Oct 31 00:29:18.800122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.799966    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" event={"ID":"e5e949a8-6bb3-4108-818f-dc99b419fdc0","Type":"ContainerDied","Data":"f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978"}
Oct 31 00:29:18.800122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.800015    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d" event={"ID":"e5e949a8-6bb3-4108-818f-dc99b419fdc0","Type":"ContainerDied","Data":"c2381d5b09dc8a6a33280c67e40a73e685bc67a5b6034ba5ef4894245b26e17a"}
Oct 31 00:29:18.800122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.800038    4466 scope.go:122] "RemoveContainer" containerID="f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978"
Oct 31 00:29:18.809918 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.809893    4466 scope.go:122] "RemoveContainer" containerID="e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"
Oct 31 00:29:18.818116 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.818093    4466 scope.go:122] "RemoveContainer" containerID="f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978"
Oct 31 00:29:18.818428 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:29:18.818406    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978\": container with ID starting with f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978 not found: ID does not exist" containerID="f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978"
Oct 31 00:29:18.818513 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.818435    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978"} err="failed to get container status \"f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978\": rpc error: code = NotFound desc = could not find container \"f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978\": container with ID starting with f1991f1cee0487483c4e863879d179d46fb3ef071a2b8201306b8f8162919978 not found: ID does not exist"
Oct 31 00:29:18.818513 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.818455    4466 scope.go:122] "RemoveContainer" containerID="e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"
Oct 31 00:29:18.818747 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:29:18.818729    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527\": container with ID starting with e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527 not found: ID does not exist" containerID="e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"
Oct 31 00:29:18.818799 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.818754    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527"} err="failed to get container status \"e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527\": rpc error: code = NotFound desc = could not find container \"e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527\": container with ID starting with e7c2b39686ed10cc4fcd9e987273877ec256be13bd95605a6020dc05bf3ca527 not found: ID does not exist"
Oct 31 00:29:18.841465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.841436    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.841465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.841459    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.841465 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.841468    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.841660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.841478    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclf6\" (UniqueName: \"kubernetes.io/projected/e5e949a8-6bb3-4108-818f-dc99b419fdc0-kube-api-access-cclf6\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.841660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.841489    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e949a8-6bb3-4108-818f-dc99b419fdc0-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:18.989731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.989607    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"]
Oct 31 00:29:18.991319 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:18.991298    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-49837c-openshift-default-566c644bb9-wb52d"]
Oct 31 00:29:20.825938 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:29:20.825907    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" path="/var/lib/kubelet/pods/e5e949a8-6bb3-4108-818f-dc99b419fdc0/volumes"
Oct 31 00:29:38.317604 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:29:38.317563    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:30:00.176711 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.176672    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"]
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177043    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6c0091c1-6953-4cee-8eb7-54926c156ea5" containerName="maas-api"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177054    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0091c1-6953-4cee-8eb7-54926c156ea5" containerName="maas-api"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177074    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177079    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177086    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177091    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177098    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177102    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177117    4466 cpu_manager.go:414] "containerMap: removing container" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177121    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177128    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b1d726f2-fb3c-40e8-9386-cd2c3493f882" containerName="revoke-keys"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177132    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d726f2-fb3c-40e8-9386-cd2c3493f882" containerName="revoke-keys"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177187    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a87bf5-30da-4901-8be0-e399f44cf476" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177197    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d726f2-fb3c-40e8-9386-cd2c3493f882" containerName="revoke-keys"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177205    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0091c1-6953-4cee-8eb7-54926c156ea5" containerName="maas-api"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177212    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0be6323-9b09-4367-a147-c5e83b1d4600" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177219    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy"
Oct 31 00:30:00.177227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.177226    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e949a8-6bb3-4108-818f-dc99b419fdc0" containerName="istio-proxy"
Oct 31 00:30:00.181502 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.181477    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"]
Oct 31 00:30:00.181621 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.181598    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.185021 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.184995    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.263789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.263754    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"]
Oct 31 00:30:00.325238 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.325201    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"]
Oct 31 00:30:00.337957 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.337919    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw9g\" (UniqueName: \"kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.338114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.338046    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzd5f\" (UniqueName: \"kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f\") pod \"maas-api-key-cleanup-29890110-ghxnl\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.338114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.338087    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-ghxnl\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.338228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.338152    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.354245 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.354202    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:30:00.438969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.438875    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-ghxnl\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.438969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.438919    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.439227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.439018    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw9g\" (UniqueName: \"kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.439227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.439082    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzd5f\" (UniqueName: \"kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f\") pod \"maas-api-key-cleanup-29890110-ghxnl\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.439773 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.439746    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-ghxnl\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.439877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.439775    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.503070 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.503039    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw9g\" (UniqueName: \"kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.503351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.503331    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzd5f\" (UniqueName: \"kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f\") pod \"maas-api-key-cleanup-29890110-ghxnl\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:00.795871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.795784    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:00.802152 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:00.802130    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:01.181741 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:30:01.181707    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced4f06f_7e59_451a_aa7e_f9c40ab18f2e.slice/crio-927cb5c4e152f71646974a043c658ed3c36aa459e499e59f23c2b2faf6a2d197 WatchSource:0}: Error finding container 927cb5c4e152f71646974a043c658ed3c36aa459e499e59f23c2b2faf6a2d197: Status 404 returned error can't find the container with id 927cb5c4e152f71646974a043c658ed3c36aa459e499e59f23c2b2faf6a2d197
Oct 31 00:30:01.189752 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:01.185559    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"]
Oct 31 00:30:01.216175 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:30:01.216145    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb839d2_eb49_40fe_9889_b8601d358b38.slice/crio-7290598fe8c8274700c378026907d378cf43088999aee3b24551f28e7ae87f80 WatchSource:0}: Error finding container 7290598fe8c8274700c378026907d378cf43088999aee3b24551f28e7ae87f80: Status 404 returned error can't find the container with id 7290598fe8c8274700c378026907d378cf43088999aee3b24551f28e7ae87f80
Oct 31 00:30:01.218592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:01.218567    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"]
Oct 31 00:30:02.003411 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.003370    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" event={"ID":"6bb839d2-eb49-40fe-9889-b8601d358b38","Type":"ContainerStarted","Data":"689c8349b4b3f9110576b9c688d318022330e71a3456803d72c2268efbbc4128"}
Oct 31 00:30:02.003656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.003441    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" event={"ID":"6bb839d2-eb49-40fe-9889-b8601d358b38","Type":"ContainerStarted","Data":"7290598fe8c8274700c378026907d378cf43088999aee3b24551f28e7ae87f80"}
Oct 31 00:30:02.004998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.004976    4466 generic.go:356] "Generic (PLEG): container finished" podID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerID="d914fd8e617ee7f08a2aeb247b347db3263667a35066f41617e282b19f1241f5" exitCode=60
Oct 31 00:30:02.005080 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.005053    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" event={"ID":"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e","Type":"ContainerDied","Data":"d914fd8e617ee7f08a2aeb247b347db3263667a35066f41617e282b19f1241f5"}
Oct 31 00:30:02.005140 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.005112    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" event={"ID":"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e","Type":"ContainerStarted","Data":"927cb5c4e152f71646974a043c658ed3c36aa459e499e59f23c2b2faf6a2d197"}
Oct 31 00:30:02.005270 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.005257    4466 scope.go:122] "RemoveContainer" containerID="d914fd8e617ee7f08a2aeb247b347db3263667a35066f41617e282b19f1241f5"
Oct 31 00:30:02.125106 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:02.124998    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" podStartSLOduration=2.124983988 podStartE2EDuration="2.124983988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.124610881 +0000 UTC m=+2907.415159198" watchObservedRunningTime="2026-10-31 00:30:02.124983988 +0000 UTC m=+2907.415532324"
Oct 31 00:30:03.011537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.011499    4466 generic.go:356] "Generic (PLEG): container finished" podID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerID="689c8349b4b3f9110576b9c688d318022330e71a3456803d72c2268efbbc4128" exitCode=60
Oct 31 00:30:03.012029 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.011568    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" event={"ID":"6bb839d2-eb49-40fe-9889-b8601d358b38","Type":"ContainerDied","Data":"689c8349b4b3f9110576b9c688d318022330e71a3456803d72c2268efbbc4128"}
Oct 31 00:30:03.012029 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.011908    4466 scope.go:122] "RemoveContainer" containerID="689c8349b4b3f9110576b9c688d318022330e71a3456803d72c2268efbbc4128"
Oct 31 00:30:03.013521 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.013495    4466 generic.go:356] "Generic (PLEG): container finished" podID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerID="b7851a22957500ffede178499b4a72ba5c7dbb3621293949ef458209c95bde4f" exitCode=60
Oct 31 00:30:03.013625 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.013580    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" event={"ID":"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e","Type":"ContainerDied","Data":"b7851a22957500ffede178499b4a72ba5c7dbb3621293949ef458209c95bde4f"}
Oct 31 00:30:03.013693 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.013658    4466 scope.go:122] "RemoveContainer" containerID="d914fd8e617ee7f08a2aeb247b347db3263667a35066f41617e282b19f1241f5"
Oct 31 00:30:03.013884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:03.013868    4466 scope.go:122] "RemoveContainer" containerID="b7851a22957500ffede178499b4a72ba5c7dbb3621293949ef458209c95bde4f"
Oct 31 00:30:03.014469 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:03.014449    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw_odh-ai-gateway-infra(ced4f06f-7e59-451a-aa7e-f9c40ab18f2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e"
Oct 31 00:30:04.021112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:04.021076    4466 generic.go:356] "Generic (PLEG): container finished" podID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerID="9cf528178a20f850f99eb1cdf72903ced0dc6cca38a34e238e1d6af1ab992198" exitCode=60
Oct 31 00:30:04.021670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:04.021157    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" event={"ID":"6bb839d2-eb49-40fe-9889-b8601d358b38","Type":"ContainerDied","Data":"9cf528178a20f850f99eb1cdf72903ced0dc6cca38a34e238e1d6af1ab992198"}
Oct 31 00:30:04.021670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:04.021234    4466 scope.go:122] "RemoveContainer" containerID="689c8349b4b3f9110576b9c688d318022330e71a3456803d72c2268efbbc4128"
Oct 31 00:30:04.021670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:04.021524    4466 scope.go:122] "RemoveContainer" containerID="9cf528178a20f850f99eb1cdf72903ced0dc6cca38a34e238e1d6af1ab992198"
Oct 31 00:30:04.021852 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:04.021800    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-ghxnl_odh-ai-gateway-infra(6bb839d2-eb49-40fe-9889-b8601d358b38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38"
Oct 31 00:30:04.023110 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:04.023095    4466 scope.go:122] "RemoveContainer" containerID="b7851a22957500ffede178499b4a72ba5c7dbb3621293949ef458209c95bde4f"
Oct 31 00:30:04.023317 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:04.023299    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw_odh-ai-gateway-infra(ced4f06f-7e59-451a-aa7e-f9c40ab18f2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e"
Oct 31 00:30:05.029047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:05.029017    4466 scope.go:122] "RemoveContainer" containerID="9cf528178a20f850f99eb1cdf72903ced0dc6cca38a34e238e1d6af1ab992198"
Oct 31 00:30:05.029458 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:05.029248    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-ghxnl_odh-ai-gateway-infra(6bb839d2-eb49-40fe-9889-b8601d358b38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38"
Oct 31 00:30:08.317654 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:08.317607    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:30:14.822944 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:14.822912    4466 scope.go:122] "RemoveContainer" containerID="b7851a22957500ffede178499b4a72ba5c7dbb3621293949ef458209c95bde4f"
Oct 31 00:30:15.820127 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:15.820098    4466 scope.go:122] "RemoveContainer" containerID="9cf528178a20f850f99eb1cdf72903ced0dc6cca38a34e238e1d6af1ab992198"
Oct 31 00:30:16.079136 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:16.079102    4466 generic.go:356] "Generic (PLEG): container finished" podID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerID="ddfd1c4b80cc760d8286b1a84e0882a9ee6f19024ee44c9c35945fc0243f3868" exitCode=60
Oct 31 00:30:16.079596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:16.079174    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" event={"ID":"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e","Type":"ContainerDied","Data":"ddfd1c4b80cc760d8286b1a84e0882a9ee6f19024ee44c9c35945fc0243f3868"}
Oct 31 00:30:16.079596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:16.079233    4466 scope.go:122] "RemoveContainer" containerID="b7851a22957500ffede178499b4a72ba5c7dbb3621293949ef458209c95bde4f"
Oct 31 00:30:16.079596 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:16.079587    4466 scope.go:122] "RemoveContainer" containerID="ddfd1c4b80cc760d8286b1a84e0882a9ee6f19024ee44c9c35945fc0243f3868"
Oct 31 00:30:16.079886 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:16.079865    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw_odh-ai-gateway-infra(ced4f06f-7e59-451a-aa7e-f9c40ab18f2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e"
Oct 31 00:30:17.085333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.085235    4466 generic.go:356] "Generic (PLEG): container finished" podID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerID="d35b2252e2204e9b1d17d2d5db737d2669a21ffa673d15ec6a9ad8a741beaa77" exitCode=60
Oct 31 00:30:17.085333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.085260    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" event={"ID":"6bb839d2-eb49-40fe-9889-b8601d358b38","Type":"ContainerDied","Data":"d35b2252e2204e9b1d17d2d5db737d2669a21ffa673d15ec6a9ad8a741beaa77"}
Oct 31 00:30:17.085901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.085333    4466 scope.go:122] "RemoveContainer" containerID="9cf528178a20f850f99eb1cdf72903ced0dc6cca38a34e238e1d6af1ab992198"
Oct 31 00:30:17.085901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.085765    4466 scope.go:122] "RemoveContainer" containerID="d35b2252e2204e9b1d17d2d5db737d2669a21ffa673d15ec6a9ad8a741beaa77"
Oct 31 00:30:17.086057 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:17.086037    4466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-ghxnl_odh-ai-gateway-infra(6bb839d2-eb49-40fe-9889-b8601d358b38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38"
Oct 31 00:30:17.214014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.213979    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"]
Oct 31 00:30:17.352792 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.352758    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:17.408909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.408876    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca\") pod \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") "
Oct 31 00:30:17.409098 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.408988    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g\" (UniqueName: \"kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g\") pod \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\" (UID: \"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e\") "
Oct 31 00:30:17.409244 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.409224    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca" pod "ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" (UID: "ced4f06f-7e59-451a-aa7e-f9c40ab18f2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:17.411054 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.411022    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g" pod "ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" (UID: "ced4f06f-7e59-451a-aa7e-f9c40ab18f2e"). InnerVolumeSpecName "kube-api-access-gpw9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:17.510324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.510289    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:17.510324 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:17.510318    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpw9g\" (UniqueName: \"kubernetes.io/projected/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e-kube-api-access-gpw9g\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:18.094262 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.094220    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"
Oct 31 00:30:18.094262 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.094252    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw" event={"ID":"ced4f06f-7e59-451a-aa7e-f9c40ab18f2e","Type":"ContainerDied","Data":"927cb5c4e152f71646974a043c658ed3c36aa459e499e59f23c2b2faf6a2d197"}
Oct 31 00:30:18.094812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.094342    4466 scope.go:122] "RemoveContainer" containerID="ddfd1c4b80cc760d8286b1a84e0882a9ee6f19024ee44c9c35945fc0243f3868"
Oct 31 00:30:18.220132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.220098    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"]
Oct 31 00:30:18.268413 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.267873    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"]
Oct 31 00:30:18.275807 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.275770    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890110-9rpfw"]
Oct 31 00:30:18.349392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.349325    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:18.420909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.420871    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f\" (UniqueName: \"kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f\") pod \"6bb839d2-eb49-40fe-9889-b8601d358b38\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") "
Oct 31 00:30:18.421124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.421007    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca\") pod \"6bb839d2-eb49-40fe-9889-b8601d358b38\" (UID: \"6bb839d2-eb49-40fe-9889-b8601d358b38\") "
Oct 31 00:30:18.421347 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.421323    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca" pod "6bb839d2-eb49-40fe-9889-b8601d358b38" (UID: "6bb839d2-eb49-40fe-9889-b8601d358b38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:18.423026 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.423001    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f" pod "6bb839d2-eb49-40fe-9889-b8601d358b38" (UID: "6bb839d2-eb49-40fe-9889-b8601d358b38"). InnerVolumeSpecName "kube-api-access-wzd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:18.522167 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.522118    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb839d2-eb49-40fe-9889-b8601d358b38-maas-api-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:18.522167 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.522157    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzd5f\" (UniqueName: \"kubernetes.io/projected/6bb839d2-eb49-40fe-9889-b8601d358b38-kube-api-access-wzd5f\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:18.826361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:18.826283    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" path="/var/lib/kubelet/pods/ced4f06f-7e59-451a-aa7e-f9c40ab18f2e/volumes"
Oct 31 00:30:19.100648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:19.100614    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"
Oct 31 00:30:19.101170 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:19.100605    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl" event={"ID":"6bb839d2-eb49-40fe-9889-b8601d358b38","Type":"ContainerDied","Data":"7290598fe8c8274700c378026907d378cf43088999aee3b24551f28e7ae87f80"}
Oct 31 00:30:19.101170 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:19.100764    4466 scope.go:122] "RemoveContainer" containerID="d35b2252e2204e9b1d17d2d5db737d2669a21ffa673d15ec6a9ad8a741beaa77"
Oct 31 00:30:19.287786 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:19.287750    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"]
Oct 31 00:30:19.294320 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:19.294291    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-ghxnl"]
Oct 31 00:30:20.826368 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:20.826332    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" path="/var/lib/kubelet/pods/6bb839d2-eb49-40fe-9889-b8601d358b38/volumes"
Oct 31 00:30:31.811816 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:31.811779    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"]
Oct 31 00:30:35.764302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:35.764270    4466 scope.go:122] "RemoveContainer" containerID="20a291f1f4c5a5d9f3c7a0a7a4b07aef0501e5fcd41c5679a55247428591635d"
Oct 31 00:30:38.316663 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:38.316603    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:30:46.831448 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:46.831389    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" containerID="cri-o://c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e" gracePeriod=45
Oct 31 00:30:46.893185 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:46.893141    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 31 00:30:47.077583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.077553    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:30:47.095536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095494    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx\" (UniqueName: \"kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx\") pod \"7b3bb64b-08ff-4946-bd36-14178156cc8d\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") "
Oct 31 00:30:47.095713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095548    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir\") pod \"7b3bb64b-08ff-4946-bd36-14178156cc8d\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") "
Oct 31 00:30:47.095713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095590    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm\") pod \"7b3bb64b-08ff-4946-bd36-14178156cc8d\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") "
Oct 31 00:30:47.095713 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095703    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache\") pod \"7b3bb64b-08ff-4946-bd36-14178156cc8d\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") "
Oct 31 00:30:47.095891 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095721    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home\") pod \"7b3bb64b-08ff-4946-bd36-14178156cc8d\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") "
Oct 31 00:30:47.095891 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095791    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs\") pod \"7b3bb64b-08ff-4946-bd36-14178156cc8d\" (UID: \"7b3bb64b-08ff-4946-bd36-14178156cc8d\") "
Oct 31 00:30:47.095891 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095871    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir" pod "7b3bb64b-08ff-4946-bd36-14178156cc8d" (UID: "7b3bb64b-08ff-4946-bd36-14178156cc8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.096056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.095971    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache" pod "7b3bb64b-08ff-4946-bd36-14178156cc8d" (UID: "7b3bb64b-08ff-4946-bd36-14178156cc8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.096056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.096012    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home" pod "7b3bb64b-08ff-4946-bd36-14178156cc8d" (UID: "7b3bb64b-08ff-4946-bd36-14178156cc8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.096163 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.096146    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.096227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.096169    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.096227 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.096186    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.097861 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.097838    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs" pod "7b3bb64b-08ff-4946-bd36-14178156cc8d" (UID: "7b3bb64b-08ff-4946-bd36-14178156cc8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:47.097861 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.097849    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm" pod "7b3bb64b-08ff-4946-bd36-14178156cc8d" (UID: "7b3bb64b-08ff-4946-bd36-14178156cc8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.097861 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.097855    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx" pod "7b3bb64b-08ff-4946-bd36-14178156cc8d" (UID: "7b3bb64b-08ff-4946-bd36-14178156cc8d"). InnerVolumeSpecName "kube-api-access-xt8dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:47.196980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.196940    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bb64b-08ff-4946-bd36-14178156cc8d-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.196980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.196976    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt8dx\" (UniqueName: \"kubernetes.io/projected/7b3bb64b-08ff-4946-bd36-14178156cc8d-kube-api-access-xt8dx\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.196980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.196990    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bb64b-08ff-4946-bd36-14178156cc8d-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.237961 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.237919    4466 generic.go:356] "Generic (PLEG): container finished" podID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerID="c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e" exitCode=0
Oct 31 00:30:47.237961 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.237961    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"
Oct 31 00:30:47.238180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.237997    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" event={"ID":"7b3bb64b-08ff-4946-bd36-14178156cc8d","Type":"ContainerDied","Data":"c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e"}
Oct 31 00:30:47.238180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.238040    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x" event={"ID":"7b3bb64b-08ff-4946-bd36-14178156cc8d","Type":"ContainerDied","Data":"99f43af7dd1cd210ec34333cc09fbdb1f55b7f2ac1646a4ba02acef8e941e8fb"}
Oct 31 00:30:47.238180 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.238064    4466 scope.go:122] "RemoveContainer" containerID="c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e"
Oct 31 00:30:47.248503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.248479    4466 scope.go:122] "RemoveContainer" containerID="c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e"
Oct 31 00:30:47.248875 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:30:47.248845    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e\": container with ID starting with c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e not found: ID does not exist" containerID="c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e"
Oct 31 00:30:47.248933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.248891    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e"} err="failed to get container status \"c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e\": rpc error: code = NotFound desc = could not find container \"c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e\": container with ID starting with c2c129acf392b71f83ca60e3f7423acfcb9705ca4dbc214eaa842cd2987c9e9e not found: ID does not exist"
Oct 31 00:30:47.409958 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.409869    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"]
Oct 31 00:30:47.415289 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:47.415261    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-w0-46b586-kserve-6c5777f6bd-89b8x"]
Oct 31 00:30:48.826154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:30:48.826123    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" path="/var/lib/kubelet/pods/7b3bb64b-08ff-4946-bd36-14178156cc8d/volumes"
Oct 31 00:31:08.316987 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:31:08.316947    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:31:10.513833 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.513799    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"]
Oct 31 00:31:10.514244 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.514004    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" podUID="08ba4b2d-ab49-4b55-8295-de4becb4f530" containerName="maas-api" containerID="cri-o://74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259" gracePeriod=30
Oct 31 00:31:10.758159 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.758135    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 31 00:31:10.928399 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.928358    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca\") pod \"08ba4b2d-ab49-4b55-8295-de4becb4f530\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") "
Oct 31 00:31:10.928617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.928409    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\") pod \"08ba4b2d-ab49-4b55-8295-de4becb4f530\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") "
Oct 31 00:31:10.928617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.928443    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs\") pod \"08ba4b2d-ab49-4b55-8295-de4becb4f530\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") "
Oct 31 00:31:10.928617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.928471    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j\" (UniqueName: \"kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j\") pod \"08ba4b2d-ab49-4b55-8295-de4becb4f530\" (UID: \"08ba4b2d-ab49-4b55-8295-de4becb4f530\") "
Oct 31 00:31:10.928874 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.928834    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca" pod "08ba4b2d-ab49-4b55-8295-de4becb4f530" (UID: "08ba4b2d-ab49-4b55-8295-de4becb4f530"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:10.930795 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.930764    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs" pod "08ba4b2d-ab49-4b55-8295-de4becb4f530" (UID: "08ba4b2d-ab49-4b55-8295-de4becb4f530"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:10.930914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.930850    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls" pod "08ba4b2d-ab49-4b55-8295-de4becb4f530" (UID: "08ba4b2d-ab49-4b55-8295-de4becb4f530"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:10.930914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:10.930861    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j" pod "08ba4b2d-ab49-4b55-8295-de4becb4f530" (UID: "08ba4b2d-ab49-4b55-8295-de4becb4f530"). InnerVolumeSpecName "kube-api-access-kx66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:11.030140 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.030100    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba4b2d-ab49-4b55-8295-de4becb4f530-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:11.030140 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.030133    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:11.030140 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.030144    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba4b2d-ab49-4b55-8295-de4becb4f530-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:11.030422 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.030153    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx66j\" (UniqueName: \"kubernetes.io/projected/08ba4b2d-ab49-4b55-8295-de4becb4f530-kube-api-access-kx66j\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:11.346172 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.346132    4466 generic.go:356] "Generic (PLEG): container finished" podID="08ba4b2d-ab49-4b55-8295-de4becb4f530" containerID="74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259" exitCode=0
Oct 31 00:31:11.346388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.346190    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"
Oct 31 00:31:11.346388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.346220    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" event={"ID":"08ba4b2d-ab49-4b55-8295-de4becb4f530","Type":"ContainerDied","Data":"74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259"}
Oct 31 00:31:11.346388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.346306    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht" event={"ID":"08ba4b2d-ab49-4b55-8295-de4becb4f530","Type":"ContainerDied","Data":"926fdae8249506a5d6a825bb4ad892932a77397da9e63b1c49f1e63433b7442a"}
Oct 31 00:31:11.346388 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.346329    4466 scope.go:122] "RemoveContainer" containerID="74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259"
Oct 31 00:31:11.355675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.355657    4466 scope.go:122] "RemoveContainer" containerID="74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259"
Oct 31 00:31:11.355970 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:31:11.355948    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259\": container with ID starting with 74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259 not found: ID does not exist" containerID="74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259"
Oct 31 00:31:11.356035 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.355984    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259"} err="failed to get container status \"74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259\": rpc error: code = NotFound desc = could not find container \"74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259\": container with ID starting with 74d06124a00d3c13660b0a2ad5b0ed63cc49591447897a34d57a463bd1be8259 not found: ID does not exist"
Oct 31 00:31:11.518200 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.518163    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"]
Oct 31 00:31:11.523085 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:11.523060    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46b586-59d64b556-5kcht"]
Oct 31 00:31:12.826334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:12.826298    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ba4b2d-ab49-4b55-8295-de4becb4f530" path="/var/lib/kubelet/pods/08ba4b2d-ab49-4b55-8295-de4becb4f530/volumes"
Oct 31 00:31:33.630600 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:33.630565    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"]
Oct 31 00:31:38.316810 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:31:38.316756    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:31:38.631933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:38.631856    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerName="payload-processing" containerID="cri-o://a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a" gracePeriod=25
Oct 31 00:31:46.293377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:46.293339    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:56.292980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:56.292942    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:58.874537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:58.874507    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:31:58.979296 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:58.979206    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh\" (UniqueName: \"kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh\") pod \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") "
Oct 31 00:31:58.979296 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:58.979267    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume\") pod \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\" (UID: \"28a7e57a-6ccb-44ab-9cb0-301c89afbae8\") "
Oct 31 00:31:58.979759 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:58.979726    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume" pod "28a7e57a-6ccb-44ab-9cb0-301c89afbae8" (UID: "28a7e57a-6ccb-44ab-9cb0-301c89afbae8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:58.981401 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:58.981376    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh" pod "28a7e57a-6ccb-44ab-9cb0-301c89afbae8" (UID: "28a7e57a-6ccb-44ab-9cb0-301c89afbae8"). InnerVolumeSpecName "kube-api-access-d49zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:59.080867 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.080830    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49zh\" (UniqueName: \"kubernetes.io/projected/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-kube-api-access-d49zh\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:59.080867 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.080864    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a7e57a-6ccb-44ab-9cb0-301c89afbae8-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:59.558327 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.558271    4466 generic.go:356] "Generic (PLEG): container finished" podID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerID="a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a" exitCode=0
Oct 31 00:31:59.558520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.558342    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"
Oct 31 00:31:59.558520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.558350    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" event={"ID":"28a7e57a-6ccb-44ab-9cb0-301c89afbae8","Type":"ContainerDied","Data":"a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a"}
Oct 31 00:31:59.558520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.558406    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm" event={"ID":"28a7e57a-6ccb-44ab-9cb0-301c89afbae8","Type":"ContainerDied","Data":"0711b5586c82b1792d656bd069bb4d4433fb288a534dfeafd6fe79a10e6464ff"}
Oct 31 00:31:59.558520 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.558436    4466 scope.go:122] "RemoveContainer" containerID="a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a"
Oct 31 00:31:59.567600 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.567581    4466 scope.go:122] "RemoveContainer" containerID="a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a"
Oct 31 00:31:59.567998 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:31:59.567977    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a\": container with ID starting with a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a not found: ID does not exist" containerID="a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a"
Oct 31 00:31:59.568063 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.568007    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a"} err="failed to get container status \"a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a\": rpc error: code = NotFound desc = could not find container \"a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a\": container with ID starting with a7b52ed3aefceaccd4d9e6e6001d5bad1fcc8ad948b201620fd30fff33e1867a not found: ID does not exist"
Oct 31 00:31:59.742554 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.742512    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"]
Oct 31 00:31:59.750459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:31:59.750429    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-37abc0-54fc77c78-vvrhm"]
Oct 31 00:32:00.539052 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:00.539017    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"]
Oct 31 00:32:00.825193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:00.825101    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" path="/var/lib/kubelet/pods/28a7e57a-6ccb-44ab-9cb0-301c89afbae8/volumes"
Oct 31 00:32:08.316685 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:32:08.316647    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:32:15.558678 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.558596    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main" containerID="cri-o://1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e" gracePeriod=45
Oct 31 00:32:15.818214 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.818149    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:32:15.834072 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834034    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs\" (UniqueName: \"kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs\") pod \"a685811a-933a-446e-bbbc-b9433d89b356\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") "
Oct 31 00:32:15.834230 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834089    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache\") pod \"a685811a-933a-446e-bbbc-b9433d89b356\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") "
Oct 31 00:32:15.834230 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834135    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt\" (UniqueName: \"kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt\") pod \"a685811a-933a-446e-bbbc-b9433d89b356\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") "
Oct 31 00:32:15.834230 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834189    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir\") pod \"a685811a-933a-446e-bbbc-b9433d89b356\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") "
Oct 31 00:32:15.834422 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834251    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home\") pod \"a685811a-933a-446e-bbbc-b9433d89b356\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") "
Oct 31 00:32:15.834422 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834334    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm\") pod \"a685811a-933a-446e-bbbc-b9433d89b356\" (UID: \"a685811a-933a-446e-bbbc-b9433d89b356\") "
Oct 31 00:32:15.834533 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834428    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache" pod "a685811a-933a-446e-bbbc-b9433d89b356" (UID: "a685811a-933a-446e-bbbc-b9433d89b356"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.834592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834507    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir" pod "a685811a-933a-446e-bbbc-b9433d89b356" (UID: "a685811a-933a-446e-bbbc-b9433d89b356"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.834686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834578    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home" pod "a685811a-933a-446e-bbbc-b9433d89b356" (UID: "a685811a-933a-446e-bbbc-b9433d89b356"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.834754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834712    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.834754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834731    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.834754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.834744    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.836479 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.836452    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm" pod "a685811a-933a-446e-bbbc-b9433d89b356" (UID: "a685811a-933a-446e-bbbc-b9433d89b356"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.836583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.836454    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs" pod "a685811a-933a-446e-bbbc-b9433d89b356" (UID: "a685811a-933a-446e-bbbc-b9433d89b356"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:15.836583 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.836516    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt" pod "a685811a-933a-446e-bbbc-b9433d89b356" (UID: "a685811a-933a-446e-bbbc-b9433d89b356"). InnerVolumeSpecName "kube-api-access-6nrnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:15.935417 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.935378    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a685811a-933a-446e-bbbc-b9433d89b356-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.935417 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.935409    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrnt\" (UniqueName: \"kubernetes.io/projected/a685811a-933a-446e-bbbc-b9433d89b356-kube-api-access-6nrnt\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.935417 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:15.935419    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a685811a-933a-446e-bbbc-b9433d89b356-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.632443 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.632405    4466 generic.go:356] "Generic (PLEG): container finished" podID="a685811a-933a-446e-bbbc-b9433d89b356" containerID="1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e" exitCode=0
Oct 31 00:32:16.632914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.632451    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"
Oct 31 00:32:16.632914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.632492    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" event={"ID":"a685811a-933a-446e-bbbc-b9433d89b356","Type":"ContainerDied","Data":"1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e"}
Oct 31 00:32:16.632914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.632541    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz" event={"ID":"a685811a-933a-446e-bbbc-b9433d89b356","Type":"ContainerDied","Data":"4e29f60e1caf2ee2b2704dffb7babbb2e546f94a2ada4fa72719781cc9f60486"}
Oct 31 00:32:16.632914 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.632562    4466 scope.go:122] "RemoveContainer" containerID="1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e"
Oct 31 00:32:16.641961 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.641941    4466 scope.go:122] "RemoveContainer" containerID="1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e"
Oct 31 00:32:16.642297 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:32:16.642259    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e\": container with ID starting with 1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e not found: ID does not exist" containerID="1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e"
Oct 31 00:32:16.642349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.642310    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e"} err="failed to get container status \"1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e\": rpc error: code = NotFound desc = could not find container \"1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e\": container with ID starting with 1da51c89808bee99379bb27e74c1d69537ac18293adf8d2bf0bbb87217bd8a6e not found: ID does not exist"
Oct 31 00:32:16.804617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.804584    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"]
Oct 31 00:32:16.856927 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:16.856885    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-2-w1-5c3374-kserve-69684d86c8-5lmmz"]
Oct 31 00:32:18.054269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:18.054226    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"]
Oct 31 00:32:18.108133 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:18.108090    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb41069-vsz5w"]
Oct 31 00:32:18.824999 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:18.824966    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a685811a-933a-446e-bbbc-b9433d89b356" path="/var/lib/kubelet/pods/a685811a-933a-446e-bbbc-b9433d89b356/volumes"
Oct 31 00:32:18.825386 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:18.825372    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a6711d-61ea-4c59-8f23-5b8e562777f7" path="/var/lib/kubelet/pods/c9a6711d-61ea-4c59-8f23-5b8e562777f7/volumes"
Oct 31 00:32:26.345955 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.345908    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"]
Oct 31 00:32:26.346404 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.346128    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" podUID="b756e4c9-eb7f-4602-986f-1cf34e213542" containerName="maas-api" containerID="cri-o://babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7" gracePeriod=30
Oct 31 00:32:26.598511 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.598447    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 31 00:32:26.675158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.675127    4466 generic.go:356] "Generic (PLEG): container finished" podID="b756e4c9-eb7f-4602-986f-1cf34e213542" containerID="babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7" exitCode=0
Oct 31 00:32:26.675361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.675188    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"
Oct 31 00:32:26.675361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.675208    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" event={"ID":"b756e4c9-eb7f-4602-986f-1cf34e213542","Type":"ContainerDied","Data":"babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7"}
Oct 31 00:32:26.675361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.675275    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5" event={"ID":"b756e4c9-eb7f-4602-986f-1cf34e213542","Type":"ContainerDied","Data":"db2f86f7c86d2c5fd74db7b2a6a012eda38c61e2dc5adc6c0cc6fe7abba3e265"}
Oct 31 00:32:26.675361 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.675306    4466 scope.go:122] "RemoveContainer" containerID="babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7"
Oct 31 00:32:26.684121 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.684095    4466 scope.go:122] "RemoveContainer" containerID="babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7"
Oct 31 00:32:26.684426 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:32:26.684406    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7\": container with ID starting with babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7 not found: ID does not exist" containerID="babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7"
Oct 31 00:32:26.684497 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.684440    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7"} err="failed to get container status \"babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7\": rpc error: code = NotFound desc = could not find container \"babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7\": container with ID starting with babaac9bb86b04bdd5a8ca16d9b936056059f0c8ecbd164c6e1ca9846c11b8e7 not found: ID does not exist"
Oct 31 00:32:26.735271 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.735239    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv\" (UniqueName: \"kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv\") pod \"b756e4c9-eb7f-4602-986f-1cf34e213542\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") "
Oct 31 00:32:26.735471 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.735296    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs\") pod \"b756e4c9-eb7f-4602-986f-1cf34e213542\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") "
Oct 31 00:32:26.735471 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.735354    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca\") pod \"b756e4c9-eb7f-4602-986f-1cf34e213542\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") "
Oct 31 00:32:26.735471 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.735392    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\") pod \"b756e4c9-eb7f-4602-986f-1cf34e213542\" (UID: \"b756e4c9-eb7f-4602-986f-1cf34e213542\") "
Oct 31 00:32:26.735760 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.735729    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca" pod "b756e4c9-eb7f-4602-986f-1cf34e213542" (UID: "b756e4c9-eb7f-4602-986f-1cf34e213542"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:26.737512 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.737485    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv" pod "b756e4c9-eb7f-4602-986f-1cf34e213542" (UID: "b756e4c9-eb7f-4602-986f-1cf34e213542"). InnerVolumeSpecName "kube-api-access-285mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:26.737617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.737496    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls" pod "b756e4c9-eb7f-4602-986f-1cf34e213542" (UID: "b756e4c9-eb7f-4602-986f-1cf34e213542"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:26.737864 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.737851    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs" pod "b756e4c9-eb7f-4602-986f-1cf34e213542" (UID: "b756e4c9-eb7f-4602-986f-1cf34e213542"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:26.836777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.836748    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285mv\" (UniqueName: \"kubernetes.io/projected/b756e4c9-eb7f-4602-986f-1cf34e213542-kube-api-access-285mv\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:26.836777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.836777    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:26.836948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.836787    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b756e4c9-eb7f-4602-986f-1cf34e213542-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:26.836948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:26.836796    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b756e4c9-eb7f-4602-986f-1cf34e213542-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:27.095516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:27.095480    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"]
Oct 31 00:32:27.107768 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:27.107722    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5c3374-cf94d4c69-6vkk5"]
Oct 31 00:32:28.825270 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:28.825186    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b756e4c9-eb7f-4602-986f-1cf34e213542" path="/var/lib/kubelet/pods/b756e4c9-eb7f-4602-986f-1cf34e213542/volumes"
Oct 31 00:32:30.019318 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019278    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"]
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019605    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019616    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019625    4466 cpu_manager.go:414] "containerMap: removing container" podUID="b756e4c9-eb7f-4602-986f-1cf34e213542" containerName="maas-api"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019645    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b756e4c9-eb7f-4602-986f-1cf34e213542" containerName="maas-api"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019658    4466 cpu_manager.go:414] "containerMap: removing container" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerName="payload-processing"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019663    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerName="payload-processing"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019678    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019682    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019689    4466 cpu_manager.go:414] "containerMap: removing container" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019693    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019702    4466 cpu_manager.go:414] "containerMap: removing container" podUID="08ba4b2d-ab49-4b55-8295-de4becb4f530" containerName="maas-api"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019707    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ba4b2d-ab49-4b55-8295-de4becb4f530" containerName="maas-api"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019715    4466 cpu_manager.go:414] "containerMap: removing container" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019720    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019727    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019731    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019744    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:30.019745 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019749    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019813    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ba4b2d-ab49-4b55-8295-de4becb4f530" containerName="maas-api"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019824    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019829    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a685811a-933a-446e-bbbc-b9433d89b356" containerName="main"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019836    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019843    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019848    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019853    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b756e4c9-eb7f-4602-986f-1cf34e213542" containerName="maas-api"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019859    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a7e57a-6ccb-44ab-9cb0-301c89afbae8" containerName="payload-processing"
Oct 31 00:32:30.020265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.019866    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3bb64b-08ff-4946-bd36-14178156cc8d" containerName="main"
Oct 31 00:32:30.024260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.024238    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.079417 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.079376    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:32:30.131683 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.131648    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5c3374-openshift-default-dockercfg-jbr44\""
Oct 31 00:32:30.134396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.134367    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"]
Oct 31 00:32:30.168568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168531    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168574    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168763 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168738    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168773    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168840    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5nx\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168874    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168893    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.168998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.168981    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.169032 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.169009    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270128    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270175    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5nx\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270220    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270250    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270305    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270332    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270407    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270441    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270551 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270504    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270749    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.270901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270837    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.271012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270904    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.271012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.270944    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.271012 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.271004    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.272806 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.272783    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.272910 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.272889    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.334225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.334191    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.335684 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.335663    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5nx\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.336431 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.336404    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:30.756315 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:32:30.756283    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683bbb93_a562_4b98_b282_2672e0fd89d5.slice/crio-f28fe3247930c2a493192dddfe713d5c3f6b7ec8cfca73afcc040e2baaff1ec5 WatchSource:0}: Error finding container f28fe3247930c2a493192dddfe713d5c3f6b7ec8cfca73afcc040e2baaff1ec5: Status 404 returned error can't find the container with id f28fe3247930c2a493192dddfe713d5c3f6b7ec8cfca73afcc040e2baaff1ec5
Oct 31 00:32:30.758392 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.758360    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:32:30.758478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.758425    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:32:30.758478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.758453    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:32:30.759097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:30.759072    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"]
Oct 31 00:32:31.701828 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:31.701793    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" event={"ID":"683bbb93-a562-4b98-b282-2672e0fd89d5","Type":"ContainerStarted","Data":"c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78"}
Oct 31 00:32:31.702271 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:31.701854    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" event={"ID":"683bbb93-a562-4b98-b282-2672e0fd89d5","Type":"ContainerStarted","Data":"f28fe3247930c2a493192dddfe713d5c3f6b7ec8cfca73afcc040e2baaff1ec5"}
Oct 31 00:32:31.881229 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:31.881161    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podStartSLOduration=2.88114765 podStartE2EDuration="2.88114765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:31.880268583 +0000 UTC m=+3057.170816901" watchObservedRunningTime="2026-10-31 00:32:31.88114765 +0000 UTC m=+3057.171695966"
Oct 31 00:32:32.348659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:32.345039    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:32.352657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:32.349302    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:32:32.352657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:32.349366    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 00:32:33.347152 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:33.347108    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:32:33.347837 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:33.347802    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 00:32:34.346662 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:34.344120    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:32:34.346662 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:34.344187    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 00:32:35.344128 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:35.344088    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:32:35.344774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:35.344733    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 00:32:36.356782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:36.356739    4466 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 00:32:36.357269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:36.356805    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 00:32:36.362592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:36.362560    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"]
Oct 31 00:32:36.362783 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:36.362762    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy" containerID="cri-o://c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78" gracePeriod=30
Oct 31 00:32:38.316595 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:32:38.316556    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:32:43.113884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.113855    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:43.210022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.209929    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.209971    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210001    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210029    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210063    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210097    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210137    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210184    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210216    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data\") pod \"683bbb93-a562-4b98-b282-2672e0fd89d5\" (UID: \"683bbb93-a562-4b98-b282-2672e0fd89d5\") "
Oct 31 00:32:43.210349 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210232    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:43.210774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210439    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:43.210774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210564    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.210774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210583    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.210774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210680    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:43.210774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210709    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:43.210980 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.210850    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:43.212293 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.212275    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:32:43.212405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.212385    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:43.212689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.212673    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:43.212736 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.212716    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx" pod "683bbb93-a562-4b98-b282-2672e0fd89d5" (UID: "683bbb93-a562-4b98-b282-2672e0fd89d5"). InnerVolumeSpecName "kube-api-access-dp5nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:43.311342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311308    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.311342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311337    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bbb93-a562-4b98-b282-2672e0fd89d5-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.311342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311346    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5nx\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-kube-api-access-dp5nx\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.311592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311356    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.311592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311365    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.311592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311373    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.311592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.311381    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bbb93-a562-4b98-b282-2672e0fd89d5-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:43.758521 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.758483    4466 generic.go:356] "Generic (PLEG): container finished" podID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerID="c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78" exitCode=0
Oct 31 00:32:43.758731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.758540    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"
Oct 31 00:32:43.758731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.758572    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" event={"ID":"683bbb93-a562-4b98-b282-2672e0fd89d5","Type":"ContainerDied","Data":"c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78"}
Oct 31 00:32:43.758731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.758629    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf" event={"ID":"683bbb93-a562-4b98-b282-2672e0fd89d5","Type":"ContainerDied","Data":"f28fe3247930c2a493192dddfe713d5c3f6b7ec8cfca73afcc040e2baaff1ec5"}
Oct 31 00:32:43.758731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.758661    4466 scope.go:122] "RemoveContainer" containerID="c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78"
Oct 31 00:32:43.763043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.763012    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c745b59c4-p8n4s"]
Oct 31 00:32:43.764000 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.763971    4466 cpu_manager.go:414] "containerMap: removing container" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:43.764000 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.763998    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:43.764186 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764056    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:43.764186 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764065    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:43.764186 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764108    4466 cpu_manager.go:414] "containerMap: removing container" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy"
Oct 31 00:32:43.764186 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764115    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy"
Oct 31 00:32:43.764371 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764253    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb839d2-eb49-40fe-9889-b8601d358b38" containerName="cleanup"
Oct 31 00:32:43.764371 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764269    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced4f06f-7e59-451a-aa7e-f9c40ab18f2e" containerName="cleanup"
Oct 31 00:32:43.764371 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.764283    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" containerName="istio-proxy"
Oct 31 00:32:43.769278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.769257    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:43.774948 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.774922    4466 scope.go:122] "RemoveContainer" containerID="c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78"
Oct 31 00:32:43.775340 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:32:43.775310    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78\": container with ID starting with c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78 not found: ID does not exist" containerID="c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78"
Oct 31 00:32:43.775423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.775353    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78"} err="failed to get container status \"c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78\": rpc error: code = NotFound desc = could not find container \"c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78\": container with ID starting with c8d831e256bcfb1eb79d637e701771e868e2d84645d8693da405c251c8216b78 not found: ID does not exist"
Oct 31 00:32:43.866456 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.866415    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zz959\""
Oct 31 00:32:43.866697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.866468    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 00:32:43.866697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.866513    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 00:32:43.866697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.866558    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:32:43.868368 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.868341    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-p8n4s"]
Oct 31 00:32:43.917421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.917384    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:43.917421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.917423    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:43.917697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.917681    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dr5v\" (UniqueName: \"kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:43.931211 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.931178    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"]
Oct 31 00:32:43.935503 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:43.935479    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-s4kvf"]
Oct 31 00:32:44.018870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.018782    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.018870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.018819    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.019069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.018910    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dr5v\" (UniqueName: \"kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.021488 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.021464    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.021552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.021464    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.083268 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.083234    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dr5v\" (UniqueName: \"kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v\") pod \"maas-controller-5c745b59c4-p8n4s\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") " pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.086051 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.086022    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:44.456950 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:32:44.456918    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fb8185_8b31_40b0_a1ca_de069aae3c09.slice/crio-62607be14ae99a730b34c830182c55036492ddac6fc5df43648322506776bd43 WatchSource:0}: Error finding container 62607be14ae99a730b34c830182c55036492ddac6fc5df43648322506776bd43: Status 404 returned error can't find the container with id 62607be14ae99a730b34c830182c55036492ddac6fc5df43648322506776bd43
Oct 31 00:32:44.458424 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.458404    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:32:44.460069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.460050    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-p8n4s"]
Oct 31 00:32:44.764209 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.764115    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-p8n4s" event={"ID":"d1fb8185-8b31-40b0-a1ca-de069aae3c09","Type":"ContainerStarted","Data":"62607be14ae99a730b34c830182c55036492ddac6fc5df43648322506776bd43"}
Oct 31 00:32:44.825923 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:44.825892    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683bbb93-a562-4b98-b282-2672e0fd89d5" path="/var/lib/kubelet/pods/683bbb93-a562-4b98-b282-2672e0fd89d5/volumes"
Oct 31 00:32:47.779623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:47.779583    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-p8n4s" event={"ID":"d1fb8185-8b31-40b0-a1ca-de069aae3c09","Type":"ContainerStarted","Data":"3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a"}
Oct 31 00:32:47.780131 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:47.779671    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:32:47.904043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:47.903985    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c745b59c4-p8n4s" podStartSLOduration=2.27322391 podStartE2EDuration="4.903971953s" totalImagesPullingTime="2.630748043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:47.902788676 +0000 UTC m=+3073.193336995" watchObservedRunningTime="2026-10-31 00:32:47.903971953 +0000 UTC m=+3073.194520269"
Oct 31 00:32:52.074136 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:52.074095    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"]
Oct 31 00:32:52.126829 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:52.126790    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30504b90-sw74j"]
Oct 31 00:32:52.825302 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:52.825265    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137f6a70-2208-4f31-9371-d59d7f02c91d" path="/var/lib/kubelet/pods/137f6a70-2208-4f31-9371-d59d7f02c91d/volumes"
Oct 31 00:32:58.785870 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:32:58.785839    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:33:00.272618 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.272581    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"]
Oct 31 00:33:00.276283 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.276265    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.325118 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.325080    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"]
Oct 31 00:33:00.328501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.328483    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.331218 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.331194    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:33:00.345478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.345452    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b8qjx\""
Oct 31 00:33:00.367426 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.367383    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.367622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.367436    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.367622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.367523    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.367622 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.367579    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.367820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.367660    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368045 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368014    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.368155 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368098    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.368215 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368195    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368224    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24sx\" (UniqueName: \"kubernetes.io/projected/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-kube-api-access-f24sx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368310    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.368377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368348    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.368415 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368388    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cx5k\" (UniqueName: \"kubernetes.io/projected/07876d43-bffb-4258-a116-fdcdb3e0da8c-kube-api-access-7cx5k\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.368466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368451    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368515 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368497    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368555    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.368611 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368586    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368615    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.368750 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.368704    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07876d43-bffb-4258-a116-fdcdb3e0da8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.378901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.378868    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"]
Oct 31 00:33:00.385216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.385189    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"]
Oct 31 00:33:00.469241 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469199    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07876d43-bffb-4258-a116-fdcdb3e0da8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469250    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469278    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469303    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469326    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469350    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469375    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469399    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469421    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469446    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24sx\" (UniqueName: \"kubernetes.io/projected/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-kube-api-access-f24sx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469487    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469515    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469556    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cx5k\" (UniqueName: \"kubernetes.io/projected/07876d43-bffb-4258-a116-fdcdb3e0da8c-kube-api-access-7cx5k\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469584    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469614    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469695    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469721    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469732    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.469940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.469770    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.470439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470013    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.470439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470056    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.470439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470079    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.470439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470238    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.470439 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470357    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.470737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470525    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.470900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.470873    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.471144 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.471120    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.471216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.471126    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07876d43-bffb-4258-a116-fdcdb3e0da8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.472129 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.472108    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.472243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.472159    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.472476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.472456    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.472537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.472481    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.533983 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.533907    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.534124 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.534078    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07876d43-bffb-4258-a116-fdcdb3e0da8c-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.534161 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.534084    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24sx\" (UniqueName: \"kubernetes.io/projected/ec7a617c-9fc5-49ec-a02f-d3e7af189f1a-kube-api-access-f24sx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-875hj\" (UID: \"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.534219 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.534194    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cx5k\" (UniqueName: \"kubernetes.io/projected/07876d43-bffb-4258-a116-fdcdb3e0da8c-kube-api-access-7cx5k\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x28pp\" (UID: \"07876d43-bffb-4258-a116-fdcdb3e0da8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:00.589085 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.589046    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:00.641476 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:00.641443    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:01.007625 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:33:01.007592    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7a617c_9fc5_49ec_a02f_d3e7af189f1a.slice/crio-22ab08529dc8ea8bbaf2edfdb1b46528b6d6e974d59c730307bf039ccc80a318 WatchSource:0}: Error finding container 22ab08529dc8ea8bbaf2edfdb1b46528b6d6e974d59c730307bf039ccc80a318: Status 404 returned error can't find the container with id 22ab08529dc8ea8bbaf2edfdb1b46528b6d6e974d59c730307bf039ccc80a318
Oct 31 00:33:01.009954 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.009914    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:33:01.010041 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.010000    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:33:01.010080 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.010040    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:33:01.011318 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.011290    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"]
Oct 31 00:33:01.059823 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:33:01.059788    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07876d43_bffb_4258_a116_fdcdb3e0da8c.slice/crio-fe31cd93d32f13cd8bf081d4c030f4e930ca8f2f5fff082ba8ba592893232ad7 WatchSource:0}: Error finding container fe31cd93d32f13cd8bf081d4c030f4e930ca8f2f5fff082ba8ba592893232ad7: Status 404 returned error can't find the container with id fe31cd93d32f13cd8bf081d4c030f4e930ca8f2f5fff082ba8ba592893232ad7
Oct 31 00:33:01.064121 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.063281    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"]
Oct 31 00:33:01.064121 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.063696    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:33:01.064121 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.063769    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:33:01.064121 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.063808    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:33:01.879081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.879030    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" event={"ID":"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a","Type":"ContainerStarted","Data":"db9c91cb80fbc467af0b7594a97629eb8310d52d143cb36d5f7e7498d10956cb"}
Oct 31 00:33:01.879731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.879703    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" event={"ID":"ec7a617c-9fc5-49ec-a02f-d3e7af189f1a","Type":"ContainerStarted","Data":"22ab08529dc8ea8bbaf2edfdb1b46528b6d6e974d59c730307bf039ccc80a318"}
Oct 31 00:33:01.880995 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.880967    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" event={"ID":"07876d43-bffb-4258-a116-fdcdb3e0da8c","Type":"ContainerStarted","Data":"87da77c6ba603b9343b05ab113be02f2bc0a7757568cb1cc2fd4f59717ea0593"}
Oct 31 00:33:01.881115 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:01.881011    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" event={"ID":"07876d43-bffb-4258-a116-fdcdb3e0da8c","Type":"ContainerStarted","Data":"fe31cd93d32f13cd8bf081d4c030f4e930ca8f2f5fff082ba8ba592893232ad7"}
Oct 31 00:33:02.060658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.059826    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podStartSLOduration=2.059781315 podStartE2EDuration="2.059781315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:02.059310074 +0000 UTC m=+3087.349858390" watchObservedRunningTime="2026-10-31 00:33:02.059781315 +0000 UTC m=+3087.350329636"
Oct 31 00:33:02.189493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.189369    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podStartSLOduration=2.189350389 podStartE2EDuration="2.189350389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:02.187444283 +0000 UTC m=+3087.477992601" watchObservedRunningTime="2026-10-31 00:33:02.189350389 +0000 UTC m=+3087.479898746"
Oct 31 00:33:02.590134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.590094    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:02.591824 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.591798    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:02.591932 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.591848    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:02.643460 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.643428    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:02.643900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.643876    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:02.644096 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:02.644073    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:03.590336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:03.590296    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:03.590797 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:03.590350    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:03.648862 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:03.648821    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:03.649156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:03.649127    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:04.590472 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:04.590439    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:04.590933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:04.590493    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:04.646168 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:04.646125    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:04.646495 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:04.646467    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:05.590377 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:05.590339    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:05.590542 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:05.590394    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:05.646664 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:05.645090    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:05.646664 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:05.645156    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:06.590148 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:06.590116    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:06.590417 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:06.590395    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:06.650084 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:06.650044    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:06.650721 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:06.650682    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:07.589748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:07.589701    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:07.589944 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:07.589769    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:07.642333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:07.642299    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:07.642519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:07.642351    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:08.318387 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:33:08.318258    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:33:08.590353 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:08.590319    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 00:33:08.590516 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:08.590390    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 00:33:08.643004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:08.642970    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 00:33:08.643191 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:08.643026    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 00:33:09.646861 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:09.646819    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:09.647396 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:09.647235    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:09.647904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:09.647884    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp"
Oct 31 00:33:09.747455 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:09.747421    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:09.747843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:09.747823    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:09.748562 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:09.748543    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj"
Oct 31 00:33:19.067009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.066929    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-p8n4s"]
Oct 31 00:33:19.067554 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.067124    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5c745b59c4-p8n4s" podUID="d1fb8185-8b31-40b0-a1ca-de069aae3c09" containerName="manager" containerID="cri-o://3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a" gracePeriod=30
Oct 31 00:33:19.319382 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.319305    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:33:19.482279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.482236    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v\" (UniqueName: \"kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v\") pod \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") "
Oct 31 00:33:19.482468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.482336    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs\") pod \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") "
Oct 31 00:33:19.482468 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.482381    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert\") pod \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\" (UID: \"d1fb8185-8b31-40b0-a1ca-de069aae3c09\") "
Oct 31 00:33:19.484959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.484922    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs" pod "d1fb8185-8b31-40b0-a1ca-de069aae3c09" (UID: "d1fb8185-8b31-40b0-a1ca-de069aae3c09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:33:19.484959 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.485061    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert" pod "d1fb8185-8b31-40b0-a1ca-de069aae3c09" (UID: "d1fb8185-8b31-40b0-a1ca-de069aae3c09"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:33:19.485422 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.485395    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v" pod "d1fb8185-8b31-40b0-a1ca-de069aae3c09" (UID: "d1fb8185-8b31-40b0-a1ca-de069aae3c09"). InnerVolumeSpecName "kube-api-access-8dr5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:19.584182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.584095    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:19.584182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.584126    4466 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1fb8185-8b31-40b0-a1ca-de069aae3c09-webhook-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:19.584182 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.584135    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dr5v\" (UniqueName: \"kubernetes.io/projected/d1fb8185-8b31-40b0-a1ca-de069aae3c09-kube-api-access-8dr5v\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:19.975435 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.975399    4466 generic.go:356] "Generic (PLEG): container finished" podID="d1fb8185-8b31-40b0-a1ca-de069aae3c09" containerID="3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a" exitCode=0
Oct 31 00:33:19.975675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.975455    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-p8n4s"
Oct 31 00:33:19.975675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.975487    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-p8n4s" event={"ID":"d1fb8185-8b31-40b0-a1ca-de069aae3c09","Type":"ContainerDied","Data":"3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a"}
Oct 31 00:33:19.975675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.975542    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-p8n4s" event={"ID":"d1fb8185-8b31-40b0-a1ca-de069aae3c09","Type":"ContainerDied","Data":"62607be14ae99a730b34c830182c55036492ddac6fc5df43648322506776bd43"}
Oct 31 00:33:19.975675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.975570    4466 scope.go:122] "RemoveContainer" containerID="3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a"
Oct 31 00:33:19.984438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.984419    4466 scope.go:122] "RemoveContainer" containerID="3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a"
Oct 31 00:33:19.984766 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:33:19.984744    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a\": container with ID starting with 3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a not found: ID does not exist" containerID="3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a"
Oct 31 00:33:19.984846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:19.984775    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a"} err="failed to get container status \"3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a\": rpc error: code = NotFound desc = could not find container \"3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a\": container with ID starting with 3ddeee7e59ed6d9fdf2430d48eb4bf1f05d6bb14b44da02b0fd9539f4c479f4a not found: ID does not exist"
Oct 31 00:33:20.147770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:20.147735    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-p8n4s"]
Oct 31 00:33:20.200070 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:20.200032    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-p8n4s"]
Oct 31 00:33:20.825178 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:20.825144    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fb8185-8b31-40b0-a1ca-de069aae3c09" path="/var/lib/kubelet/pods/d1fb8185-8b31-40b0-a1ca-de069aae3c09/volumes"
Oct 31 00:33:21.053649 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.053601    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"]
Oct 31 00:33:21.106313 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.106278    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0166c0-vs2lx"]
Oct 31 00:33:21.620872 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.620834    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c745b59c4-mhrbk"]
Oct 31 00:33:21.621279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.621189    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d1fb8185-8b31-40b0-a1ca-de069aae3c09" containerName="manager"
Oct 31 00:33:21.621279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.621200    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fb8185-8b31-40b0-a1ca-de069aae3c09" containerName="manager"
Oct 31 00:33:21.621279 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.621278    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fb8185-8b31-40b0-a1ca-de069aae3c09" containerName="manager"
Oct 31 00:33:21.627022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.627001    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.703866 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.703824    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c720be2-27ef-493b-b299-b4c0c4055e36-webhook-cert\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.704096 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.703983    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c720be2-27ef-493b-b299-b4c0c4055e36-metrics-certs\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.704096 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.704036    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfs5\" (UniqueName: \"kubernetes.io/projected/3c720be2-27ef-493b-b299-b4c0c4055e36-kube-api-access-srfs5\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.735395 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.735360    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zz959\""
Oct 31 00:33:21.735777 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.735760    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 00:33:21.736283 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.736108    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 00:33:21.736411 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.736311    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:33:21.739415 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.739379    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-mhrbk"]
Oct 31 00:33:21.805599 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.805559    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c720be2-27ef-493b-b299-b4c0c4055e36-metrics-certs\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.805820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.805609    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfs5\" (UniqueName: \"kubernetes.io/projected/3c720be2-27ef-493b-b299-b4c0c4055e36-kube-api-access-srfs5\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.805820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.805677    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c720be2-27ef-493b-b299-b4c0c4055e36-webhook-cert\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.808095 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.808072    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c720be2-27ef-493b-b299-b4c0c4055e36-webhook-cert\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.808095 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.808089    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c720be2-27ef-493b-b299-b4c0c4055e36-metrics-certs\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.870237 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.870197    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfs5\" (UniqueName: \"kubernetes.io/projected/3c720be2-27ef-493b-b299-b4c0c4055e36-kube-api-access-srfs5\") pod \"maas-controller-5c745b59c4-mhrbk\" (UID: \"3c720be2-27ef-493b-b299-b4c0c4055e36\") " pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:21.940803 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:21.940711    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:22.314778 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:33:22.314742    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c720be2_27ef_493b_b299_b4c0c4055e36.slice/crio-4afad1df85d801370da9dc4d5b424edb6e4d0f4ae794ce5488e09a8739684f6e WatchSource:0}: Error finding container 4afad1df85d801370da9dc4d5b424edb6e4d0f4ae794ce5488e09a8739684f6e: Status 404 returned error can't find the container with id 4afad1df85d801370da9dc4d5b424edb6e4d0f4ae794ce5488e09a8739684f6e
Oct 31 00:33:22.317243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:22.317188    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-mhrbk"]
Oct 31 00:33:22.826011 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:22.825977    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d726f2-fb3c-40e8-9386-cd2c3493f882" path="/var/lib/kubelet/pods/b1d726f2-fb3c-40e8-9386-cd2c3493f882/volumes"
Oct 31 00:33:22.990698 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:22.990659    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-mhrbk" event={"ID":"3c720be2-27ef-493b-b299-b4c0c4055e36","Type":"ContainerStarted","Data":"4afad1df85d801370da9dc4d5b424edb6e4d0f4ae794ce5488e09a8739684f6e"}
Oct 31 00:33:23.998002 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:23.997966    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-mhrbk" event={"ID":"3c720be2-27ef-493b-b299-b4c0c4055e36","Type":"ContainerStarted","Data":"d1cda520a711b5e94ef3a0086a574d161add5ea5bb7017f56339d188d1bf32e1"}
Oct 31 00:33:23.998429 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:23.998081    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:24.122052 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:24.122000    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c745b59c4-mhrbk" podStartSLOduration=2.215856488 podStartE2EDuration="3.121984681s" totalImagesPullingTime="906.128193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:24.12093932 +0000 UTC m=+3109.411487637" watchObservedRunningTime="2026-10-31 00:33:24.121984681 +0000 UTC m=+3109.412532999"
Oct 31 00:33:35.005413 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:35.005380    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c745b59c4-mhrbk"
Oct 31 00:33:35.823120 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:33:35.823083    4466 scope.go:122] "RemoveContainer" containerID="d041bb2242c56c4f3fac8ae836c218238935f65b6dedfc299d15cb3e05e16af3"
Oct 31 00:33:38.316130 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:33:38.316090    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:34:08.321160 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:34:08.321114    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:34:30.289101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.289063    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"]
Oct 31 00:34:30.292483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.292464    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.342125 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.342088    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"]
Oct 31 00:34:30.345582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.345557    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.387624 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387586    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.387846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387661    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.387846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387742    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3941e2c5-941a-41d5-8494-c20c46758469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.387846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387782    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.387973 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387888    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.387973 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387918    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3941e2c5-941a-41d5-8494-c20c46758469-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.387973 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387935    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcnx\" (UniqueName: \"kubernetes.io/projected/3941e2c5-941a-41d5-8494-c20c46758469-kube-api-access-mfcnx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.388078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.387987    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3941e2c5-941a-41d5-8494-c20c46758469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.388078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.388057    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.434199 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.434155    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"]
Oct 31 00:34:30.488568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488499    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.488778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488580    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.488778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488600    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.488778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488620    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.488778 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488697    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.488936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488788    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.488936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488829    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.488936 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488867    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3941e2c5-941a-41d5-8494-c20c46758469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488928    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv727\" (UniqueName: \"kubernetes.io/projected/2993c03e-0c6b-4437-b94c-b997e3059d91-kube-api-access-pv727\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.489081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.488979    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489008    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.489081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489059    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489248 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489109    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489289 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489270    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489286    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489305    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.489429 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489349    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3941e2c5-941a-41d5-8494-c20c46758469-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489429 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489405    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.489536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489443    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2993c03e-0c6b-4437-b94c-b997e3059d91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.489536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489515    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489536 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489520    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3941e2c5-941a-41d5-8494-c20c46758469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489706 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489542    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcnx\" (UniqueName: \"kubernetes.io/projected/3941e2c5-941a-41d5-8494-c20c46758469-kube-api-access-mfcnx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.489706 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.489590    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3941e2c5-941a-41d5-8494-c20c46758469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.491154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.491131    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3941e2c5-941a-41d5-8494-c20c46758469-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.492247 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.492221    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3941e2c5-941a-41d5-8494-c20c46758469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.498103 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.498079    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"]
Oct 31 00:34:30.554475 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.554401    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3941e2c5-941a-41d5-8494-c20c46758469-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.554618 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.554515    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcnx\" (UniqueName: \"kubernetes.io/projected/3941e2c5-941a-41d5-8494-c20c46758469-kube-api-access-mfcnx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-zgt8d\" (UID: \"3941e2c5-941a-41d5-8494-c20c46758469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.590422 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590380    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590434    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590457    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2993c03e-0c6b-4437-b94c-b997e3059d91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590500    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590533    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590558    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590615    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590697    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv727\" (UniqueName: \"kubernetes.io/projected/2993c03e-0c6b-4437-b94c-b997e3059d91-kube-api-access-pv727\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590802    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.590899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590873    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.591069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.590912    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.591069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.591008    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.591135 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.591086    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.591339 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.591322    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2993c03e-0c6b-4437-b94c-b997e3059d91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.592852 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.592828    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.593070 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.593055    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.606066 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.606035    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:30.655255 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.655227    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv727\" (UniqueName: \"kubernetes.io/projected/2993c03e-0c6b-4437-b94c-b997e3059d91-kube-api-access-pv727\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.655500 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.655467    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2993c03e-0c6b-4437-b94c-b997e3059d91-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-2xfw2\" (UID: \"2993c03e-0c6b-4437-b94c-b997e3059d91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:30.659212 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:30.658953    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:31.027908 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:34:31.027870    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3941e2c5_941a_41d5_8494_c20c46758469.slice/crio-ce5ba76ee823b0e383484c2d10d7210bad36e85c042705bde79890a8fac24f1f WatchSource:0}: Error finding container ce5ba76ee823b0e383484c2d10d7210bad36e85c042705bde79890a8fac24f1f: Status 404 returned error can't find the container with id ce5ba76ee823b0e383484c2d10d7210bad36e85c042705bde79890a8fac24f1f
Oct 31 00:34:31.030147 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.030117    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:34:31.030225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.030183    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:34:31.030225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.030210    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:34:31.031319 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.031296    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"]
Oct 31 00:34:31.079228 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:34:31.079196    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2993c03e_0c6b_4437_b94c_b997e3059d91.slice/crio-74cdfa3ea3c6b78cb40c7270a19a86a51dd7142e2ec66d883e0426bb3a77932e WatchSource:0}: Error finding container 74cdfa3ea3c6b78cb40c7270a19a86a51dd7142e2ec66d883e0426bb3a77932e: Status 404 returned error can't find the container with id 74cdfa3ea3c6b78cb40c7270a19a86a51dd7142e2ec66d883e0426bb3a77932e
Oct 31 00:34:31.082043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.082009    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:34:31.082213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.082064    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"]
Oct 31 00:34:31.082213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.082081    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:34:31.082213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.082129    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:34:31.294277 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.294168    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" event={"ID":"3941e2c5-941a-41d5-8494-c20c46758469","Type":"ContainerStarted","Data":"3c8040ecbece7b6bcccaca16b417306632d89645710fdd0633108779f8313587"}
Oct 31 00:34:31.294772 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.294270    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" event={"ID":"3941e2c5-941a-41d5-8494-c20c46758469","Type":"ContainerStarted","Data":"ce5ba76ee823b0e383484c2d10d7210bad36e85c042705bde79890a8fac24f1f"}
Oct 31 00:34:31.295923 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.295889    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" event={"ID":"2993c03e-0c6b-4437-b94c-b997e3059d91","Type":"ContainerStarted","Data":"726c2e1d78016d94572dbcc45f4c2fd1a277d83d2565a0c3ba0f515bdda51c30"}
Oct 31 00:34:31.296042 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.295943    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" event={"ID":"2993c03e-0c6b-4437-b94c-b997e3059d91","Type":"ContainerStarted","Data":"74cdfa3ea3c6b78cb40c7270a19a86a51dd7142e2ec66d883e0426bb3a77932e"}
Oct 31 00:34:31.421626 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.421560    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podStartSLOduration=1.421541151 podStartE2EDuration="1.421541151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:31.421039386 +0000 UTC m=+3176.711587704" watchObservedRunningTime="2026-10-31 00:34:31.421541151 +0000 UTC m=+3176.712089469"
Oct 31 00:34:31.556106 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.555960    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podStartSLOduration=1.55593914 podStartE2EDuration="1.55593914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:31.555325584 +0000 UTC m=+3176.845873902" watchObservedRunningTime="2026-10-31 00:34:31.55593914 +0000 UTC m=+3176.846487451"
Oct 31 00:34:31.606514 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.606476    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:31.661828 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:31.661735    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:32.608296 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:32.608258    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:32.608836 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:32.608338    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:32.674616 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:32.674576    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:32.674924 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:32.674894    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:33.607069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:33.607022    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:33.607264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:33.607078    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:33.662071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:33.662032    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:33.662672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:33.662621    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:34.607246 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:34.607199    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:34.607588 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:34.607545    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:34.662663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:34.660057    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:34.662663 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:34.660121    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:35.609525 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:35.609476    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:35.609883 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:35.609834    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:35.668190 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:35.668151    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:35.668845 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:35.668814    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:35.867720 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:35.864718    4466 scope.go:122] "RemoveContainer" containerID="29574877a6e2bb239d91c794fa834f269b120794a2f69f5b7d30c3a9b6464638"
Oct 31 00:34:35.896275 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:35.895846    4466 scope.go:122] "RemoveContainer" containerID="dd68b8ae9d7951a32e0e13696bca7e8005c272728ff3e46463c1fbc8f9475a36"
Oct 31 00:34:36.608660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:36.608190    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:36.608660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:36.608270    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:36.661660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:36.661053    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:36.661660 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:36.661119    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:37.606721 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:37.606674    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:37.607150 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:37.606747    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:37.659854 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:37.659815    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:37.660033 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:37.659884    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:38.317716 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:34:38.317669    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:34:38.606540 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:38.606489    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 00:34:38.606837 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:38.606551    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 00:34:38.660300 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:38.660252    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 00:34:38.660527 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:38.660329    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 00:34:40.648937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:40.648883    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:40.649598 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:40.648988    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:40.700805 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:40.700756    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:40.701020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:40.700821    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:41.691843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.691801    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-zgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:41.692481 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.692440    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d" podUID="3941e2c5-941a-41d5-8494-c20c46758469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:41.741602 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.741555    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-2xfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:41.741966 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.741920    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2" podUID="2993c03e-0c6b-4437-b94c-b997e3059d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:41.747824 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.747795    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:41.748534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.748506    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:41.749459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.749431    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-2xfw2"
Oct 31 00:34:41.980880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.980796    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:41.981131 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.981109    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:34:41.982006 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:34:41.981986    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-zgt8d"
Oct 31 00:35:08.316170 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:35:08.316132    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:35:38.316806 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:35:38.316768    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:36:08.425679 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:36:08.425606    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:36:38.317460 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:36:38.317422    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:37:08.316859 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:37:08.316814    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:37:38.317948 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:37:38.317905    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:38:08.316957 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:38:08.316917    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:38:38.316267 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:38:38.316225    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:39:08.317248 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:39:08.317205    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:39:38.497658 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:39:38.494836    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:40:08.316510 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:40:08.316470    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:40:31.164873 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.164830    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"]
Oct 31 00:40:31.168613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.168588    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.224921 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.224891    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"]
Oct 31 00:40:31.257148 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.257114    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.257337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.257255    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbn9\" (UniqueName: \"kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9\") pod \"payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.275135 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.275103    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0e6572\""
Oct 31 00:40:31.275379 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.275143    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2ac5ba16-dockercfg-fgnhw\""
Oct 31 00:40:31.321700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.321670    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"]
Oct 31 00:40:31.325078 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.325062    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.358911 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.358870    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbn9\" (UniqueName: \"kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9\") pod \"payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.359109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.358947    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscs2\" (UniqueName: \"kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2\") pod \"payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.359109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.359021    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.359220 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.359114    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.359685 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.359666    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.381412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.381373    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"]
Oct 31 00:40:31.422220 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.422143    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbn9\" (UniqueName: \"kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9\") pod \"payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.460126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.460090    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.460336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.460249    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscs2\" (UniqueName: \"kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2\") pod \"payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.460877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.460851    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.482149 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.482109    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:31.524737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.524704    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscs2\" (UniqueName: \"kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2\") pod \"payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.637433 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.637386    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:31.852208 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:40:31.852172    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1726b3c_a389_4f87_acc0_b43762e4a252.slice/crio-fa819f2ac70c05253a11ce25005590c14339886c08d7b57fb6e909438e1fb949 WatchSource:0}: Error finding container fa819f2ac70c05253a11ce25005590c14339886c08d7b57fb6e909438e1fb949: Status 404 returned error can't find the container with id fa819f2ac70c05253a11ce25005590c14339886c08d7b57fb6e909438e1fb949
Oct 31 00:40:31.854589 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:31.854563    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"]
Oct 31 00:40:32.008754 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:40:32.008723    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8ae1d1_82ae_4763_aa8a_c6ddfb324af0.slice/crio-defdfb4a6cddee83626a02481ba9296d04cba64ec56bb7ce8899a236dead2a53 WatchSource:0}: Error finding container defdfb4a6cddee83626a02481ba9296d04cba64ec56bb7ce8899a236dead2a53: Status 404 returned error can't find the container with id defdfb4a6cddee83626a02481ba9296d04cba64ec56bb7ce8899a236dead2a53
Oct 31 00:40:32.011837 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.011819    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"]
Oct 31 00:40:32.311963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.311866    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" event={"ID":"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0","Type":"ContainerStarted","Data":"fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6"}
Oct 31 00:40:32.311963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.311932    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:32.311963 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.311956    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" event={"ID":"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0","Type":"ContainerStarted","Data":"defdfb4a6cddee83626a02481ba9296d04cba64ec56bb7ce8899a236dead2a53"}
Oct 31 00:40:32.313444 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.313416    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" event={"ID":"d1726b3c-a389-4f87-acc0-b43762e4a252","Type":"ContainerStarted","Data":"4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c"}
Oct 31 00:40:32.313564 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.313463    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" event={"ID":"d1726b3c-a389-4f87-acc0-b43762e4a252","Type":"ContainerStarted","Data":"fa819f2ac70c05253a11ce25005590c14339886c08d7b57fb6e909438e1fb949"}
Oct 31 00:40:32.313564 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.313496    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:32.434909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.434854    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" podStartSLOduration=1.434839968 podStartE2EDuration="1.434839968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:32.434014567 +0000 UTC m=+3537.724562884" watchObservedRunningTime="2026-10-31 00:40:32.434839968 +0000 UTC m=+3537.725388286"
Oct 31 00:40:32.563478 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:32.563377    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" podStartSLOduration=1.563356285 podStartE2EDuration="1.563356285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:32.559168965 +0000 UTC m=+3537.849717284" watchObservedRunningTime="2026-10-31 00:40:32.563356285 +0000 UTC m=+3537.853904604"
Oct 31 00:40:38.320072 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:40:38.320032    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:40:43.321259 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:43.321227    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:40:43.321816 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:43.321309    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:40:56.174822 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.174788    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"]
Oct 31 00:40:56.178460 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.178443    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.234587 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.234555    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"]
Oct 31 00:40:56.285060 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.285017    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0e6572\"/\"openshift-service-ca.crt\""
Oct 31 00:40:56.285259 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.285062    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0e6572\"/\"kube-root-ca.crt\""
Oct 31 00:40:56.285259 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.285120    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0e6572\"/\"simulator-single-main-0e6572-kserve-self-signed-certs\""
Oct 31 00:40:56.285509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.285489    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0e6572\"/\"default-dockercfg-f2xvg\""
Oct 31 00:40:56.314204 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.314169    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.314482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.314237    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9td\" (UniqueName: \"kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.314482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.314274    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.314482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.314327    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.314482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.314411    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.314482 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.314461    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415097 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415067    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415162    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415211    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415275    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415326 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415320    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9td\" (UniqueName: \"kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415347    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415673 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415605    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415683    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.415741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.415728    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.417532 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.417508    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.417923 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.417902    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.479209 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.479128    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9td\" (UniqueName: \"kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td\") pod \"simulator-single-main-0e6572-kserve-b5676967c-8nt57\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") " pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.490442 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.490412    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:56.860708 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:40:56.860672    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04349e1e_5d8a_425f_a66a_49432fb41bdf.slice/crio-4f3b896bba16b67e704ba229b9d38e4f93c9b366d7b7b754edd01befc5fb782b WatchSource:0}: Error finding container 4f3b896bba16b67e704ba229b9d38e4f93c9b366d7b7b754edd01befc5fb782b: Status 404 returned error can't find the container with id 4f3b896bba16b67e704ba229b9d38e4f93c9b366d7b7b754edd01befc5fb782b
Oct 31 00:40:56.862973 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:56.862944    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"]
Oct 31 00:40:57.425544 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:57.425512    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" event={"ID":"04349e1e-5d8a-425f-a66a-49432fb41bdf","Type":"ContainerStarted","Data":"06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49"}
Oct 31 00:40:57.425544 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:57.425552    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" event={"ID":"04349e1e-5d8a-425f-a66a-49432fb41bdf","Type":"ContainerStarted","Data":"4f3b896bba16b67e704ba229b9d38e4f93c9b366d7b7b754edd01befc5fb782b"}
Oct 31 00:40:57.491330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:57.491301    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:40:57.547232 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:40:57.547174    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" podStartSLOduration=1.54715442 podStartE2EDuration="1.54715442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:57.546202696 +0000 UTC m=+3562.836751013" watchObservedRunningTime="2026-10-31 00:40:57.54715442 +0000 UTC m=+3562.837702745"
Oct 31 00:41:06.491335 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:41:06.491289    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:41:06.504205 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:41:06.504178    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:41:07.491675 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:41:07.491648    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:41:08.317056 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:41:08.317021    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:41:38.319295 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:41:38.319252    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:41:57.377793 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:41:57.377753    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"]
Oct 31 00:42:04.376054 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:04.376018    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"]
Oct 31 00:42:04.485328 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:04.485292    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"]
Oct 31 00:42:08.316901 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:42:08.316852    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:42:09.376491 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:09.376430    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerName="payload-processing" containerID="cri-o://fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6" gracePeriod=25
Oct 31 00:42:09.486501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:09.486460    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerName="payload-pre-processing" containerID="cri-o://4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c" gracePeriod=25
Oct 31 00:42:12.397884 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.397819    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" podUID="04349e1e-5d8a-425f-a66a-49432fb41bdf" containerName="main" containerID="cri-o://06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49" gracePeriod=45
Oct 31 00:42:12.481373 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.481334    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" podUID="04349e1e-5d8a-425f-a66a-49432fb41bdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": dial tcp 10.133.0.119:8000: connect: connection refused"
Oct 31 00:42:12.645336 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.645310    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:42:12.696251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696163    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm\") pod \"04349e1e-5d8a-425f-a66a-49432fb41bdf\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") "
Oct 31 00:42:12.696251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696201    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir\") pod \"04349e1e-5d8a-425f-a66a-49432fb41bdf\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") "
Oct 31 00:42:12.696251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696220    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td\" (UniqueName: \"kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td\") pod \"04349e1e-5d8a-425f-a66a-49432fb41bdf\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") "
Oct 31 00:42:12.696251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696244    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home\") pod \"04349e1e-5d8a-425f-a66a-49432fb41bdf\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") "
Oct 31 00:42:12.696613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696280    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache\") pod \"04349e1e-5d8a-425f-a66a-49432fb41bdf\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") "
Oct 31 00:42:12.696613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696304    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs\") pod \"04349e1e-5d8a-425f-a66a-49432fb41bdf\" (UID: \"04349e1e-5d8a-425f-a66a-49432fb41bdf\") "
Oct 31 00:42:12.696758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696600    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home" pod "04349e1e-5d8a-425f-a66a-49432fb41bdf" (UID: "04349e1e-5d8a-425f-a66a-49432fb41bdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:12.696758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696672    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache" pod "04349e1e-5d8a-425f-a66a-49432fb41bdf" (UID: "04349e1e-5d8a-425f-a66a-49432fb41bdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:12.696758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.696718    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir" pod "04349e1e-5d8a-425f-a66a-49432fb41bdf" (UID: "04349e1e-5d8a-425f-a66a-49432fb41bdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:12.698618 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.698592    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm" pod "04349e1e-5d8a-425f-a66a-49432fb41bdf" (UID: "04349e1e-5d8a-425f-a66a-49432fb41bdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:12.698749 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.698673    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs" pod "04349e1e-5d8a-425f-a66a-49432fb41bdf" (UID: "04349e1e-5d8a-425f-a66a-49432fb41bdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:42:12.698749 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.698732    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td" pod "04349e1e-5d8a-425f-a66a-49432fb41bdf" (UID: "04349e1e-5d8a-425f-a66a-49432fb41bdf"). InnerVolumeSpecName "kube-api-access-jz9td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:12.784070 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.784032    4466 generic.go:356] "Generic (PLEG): container finished" podID="04349e1e-5d8a-425f-a66a-49432fb41bdf" containerID="06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49" exitCode=0
Oct 31 00:42:12.784278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.784200    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"
Oct 31 00:42:12.784429 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.784318    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" event={"ID":"04349e1e-5d8a-425f-a66a-49432fb41bdf","Type":"ContainerDied","Data":"06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49"}
Oct 31 00:42:12.784534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.784469    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57" event={"ID":"04349e1e-5d8a-425f-a66a-49432fb41bdf","Type":"ContainerDied","Data":"4f3b896bba16b67e704ba229b9d38e4f93c9b366d7b7b754edd01befc5fb782b"}
Oct 31 00:42:12.784534 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.784517    4466 scope.go:122] "RemoveContainer" containerID="06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49"
Oct 31 00:42:12.798107 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.797957    4466 scope.go:122] "RemoveContainer" containerID="06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49"
Oct 31 00:42:12.798380 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:42:12.798356    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49\": container with ID starting with 06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49 not found: ID does not exist" containerID="06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49"
Oct 31 00:42:12.798484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798395    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49"} err="failed to get container status \"06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49\": rpc error: code = NotFound desc = could not find container \"06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49\": container with ID starting with 06246bc93496de6b47ac63ac67b3be298b420ed7ef23746192383aba36e01d49 not found: ID does not exist"
Oct 31 00:42:12.798484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798411    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:12.798484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798465    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:12.798484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798481    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9td\" (UniqueName: \"kubernetes.io/projected/04349e1e-5d8a-425f-a66a-49432fb41bdf-kube-api-access-jz9td\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:12.798697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798496    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:12.798697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798511    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04349e1e-5d8a-425f-a66a-49432fb41bdf-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:12.798697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.798526    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04349e1e-5d8a-425f-a66a-49432fb41bdf-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:12.971493 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.971407    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"]
Oct 31 00:42:12.975672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:12.975619    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0e6572/simulator-single-main-0e6572-kserve-b5676967c-8nt57"]
Oct 31 00:42:13.320301 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:13.320210    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:42:13.320466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:13.320214    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:42:14.825538 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:14.825506    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04349e1e-5d8a-425f-a66a-49432fb41bdf" path="/var/lib/kubelet/pods/04349e1e-5d8a-425f-a66a-49432fb41bdf/volumes"
Oct 31 00:42:23.319189 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:23.319104    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:42:23.319189 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:23.319139    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:42:29.620972 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.620947    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:42:29.728212 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.728190    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:42:29.751139 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.751115    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume\") pod \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") "
Oct 31 00:42:29.751243 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.751168    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2\" (UniqueName: \"kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2\") pod \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\" (UID: \"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0\") "
Oct 31 00:42:29.751497 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.751470    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume" pod "ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" (UID: "ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:29.753110 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.753090    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2" pod "ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" (UID: "ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0"). InnerVolumeSpecName "kube-api-access-rscs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:29.851974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.851953    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9\" (UniqueName: \"kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9\") pod \"d1726b3c-a389-4f87-acc0-b43762e4a252\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") "
Oct 31 00:42:29.852100 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.851999    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume\") pod \"d1726b3c-a389-4f87-acc0-b43762e4a252\" (UID: \"d1726b3c-a389-4f87-acc0-b43762e4a252\") "
Oct 31 00:42:29.852203 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.852189    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:29.852265 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.852203    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rscs2\" (UniqueName: \"kubernetes.io/projected/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0-kube-api-access-rscs2\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:29.852420 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.852399    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume" pod "d1726b3c-a389-4f87-acc0-b43762e4a252" (UID: "d1726b3c-a389-4f87-acc0-b43762e4a252"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:29.853988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.853968    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9" pod "d1726b3c-a389-4f87-acc0-b43762e4a252" (UID: "d1726b3c-a389-4f87-acc0-b43762e4a252"). InnerVolumeSpecName "kube-api-access-4rbn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:29.862941 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.862913    4466 generic.go:356] "Generic (PLEG): container finished" podID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerID="4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c" exitCode=0
Oct 31 00:42:29.863114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.863093    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" event={"ID":"d1726b3c-a389-4f87-acc0-b43762e4a252","Type":"ContainerDied","Data":"4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c"}
Oct 31 00:42:29.863191 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.863144    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv" event={"ID":"d1726b3c-a389-4f87-acc0-b43762e4a252","Type":"ContainerDied","Data":"fa819f2ac70c05253a11ce25005590c14339886c08d7b57fb6e909438e1fb949"}
Oct 31 00:42:29.863191 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.863174    4466 scope.go:122] "RemoveContainer" containerID="4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c"
Oct 31 00:42:29.863340 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.863326    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"
Oct 31 00:42:29.866591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.866565    4466 generic.go:356] "Generic (PLEG): container finished" podID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerID="fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6" exitCode=0
Oct 31 00:42:29.866721 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.866615    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"
Oct 31 00:42:29.866839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.866623    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" event={"ID":"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0","Type":"ContainerDied","Data":"fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6"}
Oct 31 00:42:29.867212 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.867167    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6" event={"ID":"ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0","Type":"ContainerDied","Data":"defdfb4a6cddee83626a02481ba9296d04cba64ec56bb7ce8899a236dead2a53"}
Oct 31 00:42:29.874039 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.874021    4466 scope.go:122] "RemoveContainer" containerID="4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c"
Oct 31 00:42:29.874333 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:42:29.874315    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c\": container with ID starting with 4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c not found: ID does not exist" containerID="4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c"
Oct 31 00:42:29.874420 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.874340    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c"} err="failed to get container status \"4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c\": rpc error: code = NotFound desc = could not find container \"4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c\": container with ID starting with 4a85daa3011cfc151e11d8d72320c53c048878c2a2a97d751f65f147a891af3c not found: ID does not exist"
Oct 31 00:42:29.874420 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.874358    4466 scope.go:122] "RemoveContainer" containerID="fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6"
Oct 31 00:42:29.881869 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.881854    4466 scope.go:122] "RemoveContainer" containerID="fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6"
Oct 31 00:42:29.882137 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:42:29.882118    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6\": container with ID starting with fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6 not found: ID does not exist" containerID="fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6"
Oct 31 00:42:29.882187 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.882144    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6"} err="failed to get container status \"fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6\": rpc error: code = NotFound desc = could not find container \"fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6\": container with ID starting with fbf36038ca49a341d8a1cd54d3c907fc3ca2b2b96bbf16394986408d64190dd6 not found: ID does not exist"
Oct 31 00:42:29.953213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.953148    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rbn9\" (UniqueName: \"kubernetes.io/projected/d1726b3c-a389-4f87-acc0-b43762e4a252-kube-api-access-4rbn9\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:29.953213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:29.953170    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1726b3c-a389-4f87-acc0-b43762e4a252-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:30.035334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:30.035305    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"]
Oct 31 00:42:30.039508 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:30.039485    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0e6572-64778475dfmdr6"]
Oct 31 00:42:30.152592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:30.152567    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"]
Oct 31 00:42:30.156803 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:30.156778    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0e6572-8468772hmv"]
Oct 31 00:42:30.825112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:30.825039    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" path="/var/lib/kubelet/pods/ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0/volumes"
Oct 31 00:42:30.825501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:42:30.825377    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" path="/var/lib/kubelet/pods/d1726b3c-a389-4f87-acc0-b43762e4a252/volumes"
Oct 31 00:42:38.315475 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:42:38.315441    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:43:08.314745 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:43:08.314706    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:43:38.315040 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:43:38.315007    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:44:08.316526 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:44:08.316488    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:44:38.315911 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:44:38.315870    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:45:08.315085 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:45:08.315044    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:45:11.023177 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023146    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"]
Oct 31 00:45:11.023559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023533    4466 cpu_manager.go:414] "containerMap: removing container" podUID="04349e1e-5d8a-425f-a66a-49432fb41bdf" containerName="main"
Oct 31 00:45:11.023559 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023546    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04349e1e-5d8a-425f-a66a-49432fb41bdf" containerName="main"
Oct 31 00:45:11.023648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023567    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerName="payload-pre-processing"
Oct 31 00:45:11.023648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023575    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerName="payload-pre-processing"
Oct 31 00:45:11.023648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023593    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerName="payload-processing"
Oct 31 00:45:11.023648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023601    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerName="payload-processing"
Oct 31 00:45:11.023776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023695    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="04349e1e-5d8a-425f-a66a-49432fb41bdf" containerName="main"
Oct 31 00:45:11.023776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023710    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1726b3c-a389-4f87-acc0-b43762e4a252" containerName="payload-pre-processing"
Oct 31 00:45:11.023776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.023720    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8ae1d1-82ae-4763-aa8a-c6ddfb324af0" containerName="payload-processing"
Oct 31 00:45:11.026570 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.026555    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.094483 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.094448    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"]
Oct 31 00:45:11.122877 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.122847    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.123036 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.122990    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zg64\" (UniqueName: \"kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64\") pod \"payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.132909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.132885    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e98014\""
Oct 31 00:45:11.133046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.132946    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0e97290e-dockercfg-p7jl6\""
Oct 31 00:45:11.188590 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.188568    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"]
Oct 31 00:45:11.191876 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.191862    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.223950 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.223926    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zg64\" (UniqueName: \"kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64\") pod \"payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.224046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.223984    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.224612 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.224593    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.249019 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.248994    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"]
Oct 31 00:45:11.287221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.287176    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zg64\" (UniqueName: \"kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64\") pod \"payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.324573 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.324546    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.324689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.324614    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvrg\" (UniqueName: \"kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg\") pod \"payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.338451 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.338430    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:11.425486 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.425455    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.425676 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.425551    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvrg\" (UniqueName: \"kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg\") pod \"payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.426261 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.426241    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.489226 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.489196    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvrg\" (UniqueName: \"kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg\") pod \"payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.503917 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.503893    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:11.703590 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:45:11.703555    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc640e49a_f04b_4be6_bd4f_820997401132.slice/crio-6341bd1f77c1c09a4fba69debce61665cc23c7c4d8162e3ef7f9ca2b31b1291b WatchSource:0}: Error finding container 6341bd1f77c1c09a4fba69debce61665cc23c7c4d8162e3ef7f9ca2b31b1291b: Status 404 returned error can't find the container with id 6341bd1f77c1c09a4fba69debce61665cc23c7c4d8162e3ef7f9ca2b31b1291b
Oct 31 00:45:11.705986 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.705963    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"]
Oct 31 00:45:11.869789 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:45:11.869761    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247c3bcf_9cd9_4b96_ba0a_1bde7ecd4d66.slice/crio-b0e2999c52ceb22678bc100f9411298f5a342fc6bb13ca5e114a9d2027cd18fd WatchSource:0}: Error finding container b0e2999c52ceb22678bc100f9411298f5a342fc6bb13ca5e114a9d2027cd18fd: Status 404 returned error can't find the container with id b0e2999c52ceb22678bc100f9411298f5a342fc6bb13ca5e114a9d2027cd18fd
Oct 31 00:45:11.872298 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:11.872276    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"]
Oct 31 00:45:12.629692 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.629651    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" event={"ID":"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66","Type":"ContainerStarted","Data":"fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb"}
Oct 31 00:45:12.630154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.629720    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" event={"ID":"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66","Type":"ContainerStarted","Data":"b0e2999c52ceb22678bc100f9411298f5a342fc6bb13ca5e114a9d2027cd18fd"}
Oct 31 00:45:12.630154 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.629751    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:12.631184 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.631161    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" event={"ID":"c640e49a-f04b-4be6-bd4f-820997401132","Type":"ContainerStarted","Data":"7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8"}
Oct 31 00:45:12.631278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.631200    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" event={"ID":"c640e49a-f04b-4be6-bd4f-820997401132","Type":"ContainerStarted","Data":"6341bd1f77c1c09a4fba69debce61665cc23c7c4d8162e3ef7f9ca2b31b1291b"}
Oct 31 00:45:12.631278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.631220    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:12.750593 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.750553    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" podStartSLOduration=1.750538111 podStartE2EDuration="1.750538111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:12.750282599 +0000 UTC m=+3818.040830917" watchObservedRunningTime="2026-10-31 00:45:12.750538111 +0000 UTC m=+3818.041086429"
Oct 31 00:45:12.871993 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:12.871940    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" podStartSLOduration=2.871921193 podStartE2EDuration="2.871921193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:12.87005635 +0000 UTC m=+3818.160604670" watchObservedRunningTime="2026-10-31 00:45:12.871921193 +0000 UTC m=+3818.162469510"
Oct 31 00:45:23.638448 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:23.638375    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:45:23.638820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:23.638528    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:45:35.330411 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.330378    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"]
Oct 31 00:45:35.334001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.333985    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.389625 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.389595    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"]
Oct 31 00:45:35.440082 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.440056    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e98014\"/\"default-dockercfg-tgxzn\""
Oct 31 00:45:35.440220 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.440084    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e98014\"/\"openshift-service-ca.crt\""
Oct 31 00:45:35.440220 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.440094    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e98014\"/\"kube-root-ca.crt\""
Oct 31 00:45:35.441245 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.441228    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e98014\"/\"simulator-single-main-e98014-kserve-self-signed-certs\""
Oct 31 00:45:35.453673 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.453650    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.453738 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.453706    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jm2c\" (UniqueName: \"kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.453785 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.453768    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.453847 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.453832    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.453883 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.453861    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.453965 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.453953    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.554998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.554968    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555009    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555056    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555095    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555155    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jm2c\" (UniqueName: \"kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555193 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555178    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555453 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555430    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555517 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555461    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.555517 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.555509    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.557363 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.557335    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.557601 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.557587    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.618543 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.618509    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jm2c\" (UniqueName: \"kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c\") pod \"simulator-single-main-e98014-kserve-65d59b9857-6m955\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") " pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:35.645096 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:35.645067    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:36.011507 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:45:36.011475    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b5ff521_32ea_4d01_ad6a_fafb8c4d28e4.slice/crio-55b4f7907b7932b928c04c627777e2b0265cefd865e47e6a2cc0cc9e48b8f811 WatchSource:0}: Error finding container 55b4f7907b7932b928c04c627777e2b0265cefd865e47e6a2cc0cc9e48b8f811: Status 404 returned error can't find the container with id 55b4f7907b7932b928c04c627777e2b0265cefd865e47e6a2cc0cc9e48b8f811
Oct 31 00:45:36.014481 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:36.014462    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"]
Oct 31 00:45:36.741523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:36.741489    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955" event={"ID":"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4","Type":"ContainerStarted","Data":"dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029"}
Oct 31 00:45:36.741523 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:36.741527    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955" event={"ID":"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4","Type":"ContainerStarted","Data":"55b4f7907b7932b928c04c627777e2b0265cefd865e47e6a2cc0cc9e48b8f811"}
Oct 31 00:45:36.862178 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:36.862124    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955" podStartSLOduration=1.862112719 podStartE2EDuration="1.862112719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:36.862081217 +0000 UTC m=+3842.152629534" watchObservedRunningTime="2026-10-31 00:45:36.862112719 +0000 UTC m=+3842.152661033"
Oct 31 00:45:37.645273 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:37.645242    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:38.315159 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:45:38.315123    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:45:45.646059 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:45.646021    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:45.658074 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:45.658047    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:45:45.792156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:45:45.792130    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:46:08.316476 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:46:08.316434    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:46:37.135743 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:37.135709    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"]
Oct 31 00:46:38.316709 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:46:38.316672    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:46:44.151654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:44.151604    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"]
Oct 31 00:46:44.254353 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:44.254320    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"]
Oct 31 00:46:49.152127 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:49.152007    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerName="payload-processing" containerID="cri-o://fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb" gracePeriod=25
Oct 31 00:46:49.255461 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:49.255411    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" podUID="c640e49a-f04b-4be6-bd4f-820997401132" containerName="payload-pre-processing" containerID="cri-o://7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8" gracePeriod=25
Oct 31 00:46:52.154743 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.154709    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955" podUID="7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" containerName="main" containerID="cri-o://dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029" gracePeriod=45
Oct 31 00:46:52.397386 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.397360    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:46:52.456340 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456283    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home\") pod \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") "
Oct 31 00:46:52.456474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456348    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs\") pod \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") "
Oct 31 00:46:52.456474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456414    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir\") pod \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") "
Oct 31 00:46:52.456474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456445    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache\") pod \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") "
Oct 31 00:46:52.456670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456479    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm\") pod \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") "
Oct 31 00:46:52.456670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456553    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home" pod "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" (UID: "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:52.456670 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456609    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c\" (UniqueName: \"kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c\") pod \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\" (UID: \"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4\") "
Oct 31 00:46:52.456844 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456665    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir" pod "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" (UID: "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:52.456844 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456747    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache" pod "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" (UID: "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:52.457018 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.456994    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:52.457141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.457020    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:52.457141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.457034    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:52.458589 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.458567    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs" pod "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" (UID: "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:52.458700 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.458654    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm" pod "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" (UID: "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:46:52.458746 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.458716    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c" pod "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" (UID: "7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4"). InnerVolumeSpecName "kube-api-access-4jm2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:52.558174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.558140    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:52.558174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.558169    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jm2c\" (UniqueName: \"kubernetes.io/projected/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-kube-api-access-4jm2c\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:52.558174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:52.558181    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:53.077211 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.077177    4466 generic.go:356] "Generic (PLEG): container finished" podID="7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" containerID="dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029" exitCode=0
Oct 31 00:46:53.077375 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.077223    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"
Oct 31 00:46:53.077375 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.077243    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955" event={"ID":"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4","Type":"ContainerDied","Data":"dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029"}
Oct 31 00:46:53.077375 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.077274    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955" event={"ID":"7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4","Type":"ContainerDied","Data":"55b4f7907b7932b928c04c627777e2b0265cefd865e47e6a2cc0cc9e48b8f811"}
Oct 31 00:46:53.077375 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.077290    4466 scope.go:122] "RemoveContainer" containerID="dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029"
Oct 31 00:46:53.088665 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.088645    4466 scope.go:122] "RemoveContainer" containerID="dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029"
Oct 31 00:46:53.088936 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:46:53.088912    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029\": container with ID starting with dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029 not found: ID does not exist" containerID="dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029"
Oct 31 00:46:53.088977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.088946    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029"} err="failed to get container status \"dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029\": rpc error: code = NotFound desc = could not find container \"dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029\": container with ID starting with dc93bb4dd4b4b84de053c81839e07ac4e91724375c3eaf853d2a6751cffca029 not found: ID does not exist"
Oct 31 00:46:53.247960 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.247934    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"]
Oct 31 00:46:53.253697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.253676    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e98014/simulator-single-main-e98014-kserve-65d59b9857-6m955"]
Oct 31 00:46:53.637287 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.637254    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:46:53.637467 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:53.637261    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" podUID="c640e49a-f04b-4be6-bd4f-820997401132" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:46:54.825765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:46:54.825733    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" path="/var/lib/kubelet/pods/7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4/volumes"
Oct 31 00:47:03.516871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.516835    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"]
Oct 31 00:47:03.517321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.517210    4466 cpu_manager.go:414] "containerMap: removing container" podUID="7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" containerName="main"
Oct 31 00:47:03.517321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.517221    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" containerName="main"
Oct 31 00:47:03.517321 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.517283    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5ff521-32ea-4d01-ad6a-fafb8c4d28e4" containerName="main"
Oct 31 00:47:03.521854 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.521838    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.576105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.576077    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"]
Oct 31 00:47:03.626823 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.626796    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d40847c7-kserve-self-signed-certs\""
Oct 31 00:47:03.636898 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.636870    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:03.637050 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.636875    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" podUID="c640e49a-f04b-4be6-bd4f-820997401132" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:03.666708 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.666673    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.666878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.666810    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.666878 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.666851    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.667001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.666981    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.667057 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.667041    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.667113 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.667088    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwl7\" (UniqueName: \"kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.767834 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.767756    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.767834 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.767789    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.767834 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.767834    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.768130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.767864    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.768130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.767879    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwl7\" (UniqueName: \"kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.768130 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.767922    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.768281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.768198    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.768281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.768227    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.768354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.768311    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.770211 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.770190    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.770380 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.770364    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.831454 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.831425    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwl7\" (UniqueName: \"kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7\") pod \"e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") " pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:03.833122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:03.833100    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:04.201006 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:47:04.200975    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd982356_7e6e_4d17_88c4_059bac1f55f0.slice/crio-29ba447cfd5f454f4de27f7e50ad79d35b7f1d426364cb982a28b43be5a21efb WatchSource:0}: Error finding container 29ba447cfd5f454f4de27f7e50ad79d35b7f1d426364cb982a28b43be5a21efb: Status 404 returned error can't find the container with id 29ba447cfd5f454f4de27f7e50ad79d35b7f1d426364cb982a28b43be5a21efb
Oct 31 00:47:04.204001 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:04.203980    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"]
Oct 31 00:47:05.131535 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:05.131500    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml" event={"ID":"cd982356-7e6e-4d17-88c4-059bac1f55f0","Type":"ContainerStarted","Data":"b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c"}
Oct 31 00:47:05.131931 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:05.131544    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml" event={"ID":"cd982356-7e6e-4d17-88c4-059bac1f55f0","Type":"ContainerStarted","Data":"29ba447cfd5f454f4de27f7e50ad79d35b7f1d426364cb982a28b43be5a21efb"}
Oct 31 00:47:05.253979 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:05.253929    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml" podStartSLOduration=2.253916307 podStartE2EDuration="2.253916307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:05.25340855 +0000 UTC m=+3930.543956869" watchObservedRunningTime="2026-10-31 00:47:05.253916307 +0000 UTC m=+3930.544464624"
Oct 31 00:47:05.833345 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:05.833315    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:08.316400 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:47:08.316360    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:47:09.517289 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.517254    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:47:09.627592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.626916    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume\") pod \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") "
Oct 31 00:47:09.627592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.627041    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg\" (UniqueName: \"kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg\") pod \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\" (UID: \"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66\") "
Oct 31 00:47:09.627592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.627316    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume" pod "247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" (UID: "247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:47:09.627592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.627392    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:09.759192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.732773    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg" pod "247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" (UID: "247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66"). InnerVolumeSpecName "kube-api-access-gvvrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:47:09.759192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.734143    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvvrg\" (UniqueName: \"kubernetes.io/projected/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66-kube-api-access-gvvrg\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:09.912657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.912178    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:47:09.988658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.988182    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64\" (UniqueName: \"kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64\") pod \"c640e49a-f04b-4be6-bd4f-820997401132\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") "
Oct 31 00:47:09.988658 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:09.988257    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume\") pod \"c640e49a-f04b-4be6-bd4f-820997401132\" (UID: \"c640e49a-f04b-4be6-bd4f-820997401132\") "
Oct 31 00:47:10.003106 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.003063    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64" pod "c640e49a-f04b-4be6-bd4f-820997401132" (UID: "c640e49a-f04b-4be6-bd4f-820997401132"). InnerVolumeSpecName "kube-api-access-4zg64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:47:10.008389 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.006862    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume" pod "c640e49a-f04b-4be6-bd4f-820997401132" (UID: "c640e49a-f04b-4be6-bd4f-820997401132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:47:10.093470 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.093251    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zg64\" (UniqueName: \"kubernetes.io/projected/c640e49a-f04b-4be6-bd4f-820997401132-kube-api-access-4zg64\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:10.093470 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.093282    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640e49a-f04b-4be6-bd4f-820997401132-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:10.276655 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.276471    4466 generic.go:356] "Generic (PLEG): container finished" podID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerID="fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb" exitCode=0
Oct 31 00:47:10.276854 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.276574    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"
Oct 31 00:47:10.280655 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.276592    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" event={"ID":"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66","Type":"ContainerDied","Data":"fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb"}
Oct 31 00:47:10.280655 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.277385    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2" event={"ID":"247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66","Type":"ContainerDied","Data":"b0e2999c52ceb22678bc100f9411298f5a342fc6bb13ca5e114a9d2027cd18fd"}
Oct 31 00:47:10.280655 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.277429    4466 scope.go:122] "RemoveContainer" containerID="fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb"
Oct 31 00:47:10.291656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.286505    4466 generic.go:356] "Generic (PLEG): container finished" podID="c640e49a-f04b-4be6-bd4f-820997401132" containerID="7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8" exitCode=0
Oct 31 00:47:10.291656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.286570    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" event={"ID":"c640e49a-f04b-4be6-bd4f-820997401132","Type":"ContainerDied","Data":"7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8"}
Oct 31 00:47:10.291656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.286607    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h" event={"ID":"c640e49a-f04b-4be6-bd4f-820997401132","Type":"ContainerDied","Data":"6341bd1f77c1c09a4fba69debce61665cc23c7c4d8162e3ef7f9ca2b31b1291b"}
Oct 31 00:47:10.291656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.286689    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"
Oct 31 00:47:10.356659 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.355767    4466 scope.go:122] "RemoveContainer" containerID="fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb"
Oct 31 00:47:10.359148 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:47:10.358993    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb\": container with ID starting with fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb not found: ID does not exist" containerID="fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb"
Oct 31 00:47:10.359148 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.359032    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb"} err="failed to get container status \"fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb\": rpc error: code = NotFound desc = could not find container \"fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb\": container with ID starting with fb5553aa1ac63d9881e639210b785162dc67d1f85c9a69b5f50310b49024c5cb not found: ID does not exist"
Oct 31 00:47:10.359148 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.359056    4466 scope.go:122] "RemoveContainer" containerID="7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8"
Oct 31 00:47:10.455526 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.455243    4466 scope.go:122] "RemoveContainer" containerID="7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8"
Oct 31 00:47:10.455839 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:47:10.455744    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8\": container with ID starting with 7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8 not found: ID does not exist" containerID="7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8"
Oct 31 00:47:10.455839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.455781    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8"} err="failed to get container status \"7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8\": rpc error: code = NotFound desc = could not find container \"7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8\": container with ID starting with 7c8afc9bf579112c1266463c121fc4a94ef40cadcd9ad1500b52332af45972a8 not found: ID does not exist"
Oct 31 00:47:10.517397 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.517367    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"]
Oct 31 00:47:10.522782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.522759    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e98014-6989c9wz5h"]
Oct 31 00:47:10.647558 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.647457    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"]
Oct 31 00:47:10.650677 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.650651    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e98014-54d9bccbbjcft2"]
Oct 31 00:47:10.705899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.705858    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:47:10.706141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.706100    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:10.805782 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.805749    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:47:10.806026 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.805999    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:10.863657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.862847    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" path="/var/lib/kubelet/pods/247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66/volumes"
Oct 31 00:47:10.863657 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:10.863394    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c640e49a-f04b-4be6-bd4f-820997401132" path="/var/lib/kubelet/pods/c640e49a-f04b-4be6-bd4f-820997401132/volumes"
Oct 31 00:47:13.833897 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:13.833869    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:13.845705 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:13.845687    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:14.331250 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:14.331220    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:47:38.315892 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:47:38.315857    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:47:47.922381 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922345    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"]
Oct 31 00:47:47.922794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922706    4466 cpu_manager.go:414] "containerMap: removing container" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerName="payload-processing"
Oct 31 00:47:47.922794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922717    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerName="payload-processing"
Oct 31 00:47:47.922794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922729    4466 cpu_manager.go:414] "containerMap: removing container" podUID="c640e49a-f04b-4be6-bd4f-820997401132" containerName="payload-pre-processing"
Oct 31 00:47:47.922794 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922735    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c640e49a-f04b-4be6-bd4f-820997401132" containerName="payload-pre-processing"
Oct 31 00:47:47.922932 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922813    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="247c3bcf-9cd9-4b96-ba0a-1bde7ecd4d66" containerName="payload-processing"
Oct 31 00:47:47.922932 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.922821    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="c640e49a-f04b-4be6-bd4f-820997401132" containerName="payload-pre-processing"
Oct 31 00:47:47.925784 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:47.925768    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.032855 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.032818    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d40847c7-kserve-self-signed-certs\""
Oct 31 00:47:48.034394 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.034373    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"]
Oct 31 00:47:48.075446 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.075404    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.075770 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.075736    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.075890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.075803    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.075890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.075829    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.075890 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.075857    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqkd\" (UniqueName: \"kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.076179 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.076154    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177054    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177093    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177123    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177140    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177158    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqkd\" (UniqueName: \"kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177216    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177447    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177511    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.177580 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.177527    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.179456 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.179431    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.179652 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.179624    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.240501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.240473    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqkd\" (UniqueName: \"kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd\") pod \"e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") " pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.536925 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.536842    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:48.906844 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:47:48.906812    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9314b72_0654_4aa1_a080_0d12dd89b0df.slice/crio-0fdfd0ea83129f33ef117d71242e084009a212a39409c9f58d9db87c5ffa3023 WatchSource:0}: Error finding container 0fdfd0ea83129f33ef117d71242e084009a212a39409c9f58d9db87c5ffa3023: Status 404 returned error can't find the container with id 0fdfd0ea83129f33ef117d71242e084009a212a39409c9f58d9db87c5ffa3023
Oct 31 00:47:48.909613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:48.909589    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"]
Oct 31 00:47:49.473254 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:49.473222    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98" event={"ID":"d9314b72-0654-4aa1-a080-0d12dd89b0df","Type":"ContainerStarted","Data":"96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49"}
Oct 31 00:47:49.473669 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:49.473265    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98" event={"ID":"d9314b72-0654-4aa1-a080-0d12dd89b0df","Type":"ContainerStarted","Data":"0fdfd0ea83129f33ef117d71242e084009a212a39409c9f58d9db87c5ffa3023"}
Oct 31 00:47:49.536997 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:49.536976    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:49.596282 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:49.596243    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98" podStartSLOduration=2.596231226 podStartE2EDuration="2.596231226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:49.595724809 +0000 UTC m=+3974.886273137" watchObservedRunningTime="2026-10-31 00:47:49.596231226 +0000 UTC m=+3974.886779544"
Oct 31 00:47:55.698779 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:55.698724    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:47:55.699438 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:55.699389    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:55.803841 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:55.803718    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-875hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:47:55.803841 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:55.803781    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-875hj" podUID="ec7a617c-9fc5-49ec-a02f-d3e7af189f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:47:58.537702 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:58.537665    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:58.550139 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:58.550110    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:47:59.543376 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:47:59.543342    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:48:08.316307 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:48:08.316255    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:48:32.419661 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:32.419615    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"]
Oct 31 00:48:37.201933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:37.201894    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"]
Oct 31 00:48:38.414421 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:48:38.414334    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:48:47.438348 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.438302    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml" podUID="cd982356-7e6e-4d17-88c4-059bac1f55f0" containerName="main" containerID="cri-o://b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c" gracePeriod=45
Oct 31 00:48:47.684565 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.684540    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:48:47.729293 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729216    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache\") pod \"cd982356-7e6e-4d17-88c4-059bac1f55f0\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") "
Oct 31 00:48:47.729293 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729262    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7\" (UniqueName: \"kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7\") pod \"cd982356-7e6e-4d17-88c4-059bac1f55f0\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") "
Oct 31 00:48:47.729501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729294    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm\") pod \"cd982356-7e6e-4d17-88c4-059bac1f55f0\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") "
Oct 31 00:48:47.729501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729390    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache" pod "cd982356-7e6e-4d17-88c4-059bac1f55f0" (UID: "cd982356-7e6e-4d17-88c4-059bac1f55f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:47.729501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729414    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs\") pod \"cd982356-7e6e-4d17-88c4-059bac1f55f0\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") "
Oct 31 00:48:47.729501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729454    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home\") pod \"cd982356-7e6e-4d17-88c4-059bac1f55f0\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") "
Oct 31 00:48:47.729820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729505    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir\") pod \"cd982356-7e6e-4d17-88c4-059bac1f55f0\" (UID: \"cd982356-7e6e-4d17-88c4-059bac1f55f0\") "
Oct 31 00:48:47.729820 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729733    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home" pod "cd982356-7e6e-4d17-88c4-059bac1f55f0" (UID: "cd982356-7e6e-4d17-88c4-059bac1f55f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:47.729934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729858    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:47.729934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729904    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:47.730028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.729941    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir" pod "cd982356-7e6e-4d17-88c4-059bac1f55f0" (UID: "cd982356-7e6e-4d17-88c4-059bac1f55f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:47.731604 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.731583    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs" pod "cd982356-7e6e-4d17-88c4-059bac1f55f0" (UID: "cd982356-7e6e-4d17-88c4-059bac1f55f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:47.731722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.731609    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7" pod "cd982356-7e6e-4d17-88c4-059bac1f55f0" (UID: "cd982356-7e6e-4d17-88c4-059bac1f55f0"). InnerVolumeSpecName "kube-api-access-wlwl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:47.731722 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.731681    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm" pod "cd982356-7e6e-4d17-88c4-059bac1f55f0" (UID: "cd982356-7e6e-4d17-88c4-059bac1f55f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:47.830798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.830770    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd982356-7e6e-4d17-88c4-059bac1f55f0-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:47.830798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.830794    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:47.830798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.830804    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlwl7\" (UniqueName: \"kubernetes.io/projected/cd982356-7e6e-4d17-88c4-059bac1f55f0-kube-api-access-wlwl7\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:47.831007 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.830813    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd982356-7e6e-4d17-88c4-059bac1f55f0-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:47.873748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.873718    4466 generic.go:356] "Generic (PLEG): container finished" podID="cd982356-7e6e-4d17-88c4-059bac1f55f0" containerID="b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c" exitCode=0
Oct 31 00:48:47.873892 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.873764    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"
Oct 31 00:48:47.873892 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.873808    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml" event={"ID":"cd982356-7e6e-4d17-88c4-059bac1f55f0","Type":"ContainerDied","Data":"b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c"}
Oct 31 00:48:47.873892 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.873852    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml" event={"ID":"cd982356-7e6e-4d17-88c4-059bac1f55f0","Type":"ContainerDied","Data":"29ba447cfd5f454f4de27f7e50ad79d35b7f1d426364cb982a28b43be5a21efb"}
Oct 31 00:48:47.873892 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.873882    4466 scope.go:122] "RemoveContainer" containerID="b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c"
Oct 31 00:48:47.883134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.883116    4466 scope.go:122] "RemoveContainer" containerID="b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c"
Oct 31 00:48:47.883386 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:48:47.883365    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c\": container with ID starting with b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c not found: ID does not exist" containerID="b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c"
Oct 31 00:48:47.883434 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:47.883395    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c"} err="failed to get container status \"b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c\": rpc error: code = NotFound desc = could not find container \"b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c\": container with ID starting with b69f16366821e4f420ead8ca65ce78faded5a878946a5c0a049d0a1616a2f40c not found: ID does not exist"
Oct 31 00:48:48.044687 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:48.044613    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"]
Oct 31 00:48:48.048584 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:48.048564    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d40847c7-kserve-56886d8b5-slqml"]
Oct 31 00:48:48.826568 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:48.826536    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd982356-7e6e-4d17-88c4-059bac1f55f0" path="/var/lib/kubelet/pods/cd982356-7e6e-4d17-88c4-059bac1f55f0/volumes"
Oct 31 00:48:52.233864 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.233805    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98" podUID="d9314b72-0654-4aa1-a080-0d12dd89b0df" containerName="main" containerID="cri-o://96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49" gracePeriod=45
Oct 31 00:48:52.490871 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.490808    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:48:52.575810 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.575784    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm\") pod \"d9314b72-0654-4aa1-a080-0d12dd89b0df\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") "
Oct 31 00:48:52.575982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.575828    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir\") pod \"d9314b72-0654-4aa1-a080-0d12dd89b0df\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") "
Oct 31 00:48:52.575982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.575853    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd\" (UniqueName: \"kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd\") pod \"d9314b72-0654-4aa1-a080-0d12dd89b0df\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") "
Oct 31 00:48:52.575982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.575895    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache\") pod \"d9314b72-0654-4aa1-a080-0d12dd89b0df\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") "
Oct 31 00:48:52.575982 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.575944    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs\") pod \"d9314b72-0654-4aa1-a080-0d12dd89b0df\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") "
Oct 31 00:48:52.576213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.575994    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home\") pod \"d9314b72-0654-4aa1-a080-0d12dd89b0df\" (UID: \"d9314b72-0654-4aa1-a080-0d12dd89b0df\") "
Oct 31 00:48:52.576213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.576089    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir" pod "d9314b72-0654-4aa1-a080-0d12dd89b0df" (UID: "d9314b72-0654-4aa1-a080-0d12dd89b0df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:52.576213 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.576140    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache" pod "d9314b72-0654-4aa1-a080-0d12dd89b0df" (UID: "d9314b72-0654-4aa1-a080-0d12dd89b0df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:52.576374 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.576278    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home" pod "d9314b72-0654-4aa1-a080-0d12dd89b0df" (UID: "d9314b72-0654-4aa1-a080-0d12dd89b0df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:52.576432 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.576377    4466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-tmp-dir\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.576432 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.576389    4466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-model-cache\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.576432 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.576397    4466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-home\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.577912 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.577888    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs" pod "d9314b72-0654-4aa1-a080-0d12dd89b0df" (UID: "d9314b72-0654-4aa1-a080-0d12dd89b0df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:52.578014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.577997    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm" pod "d9314b72-0654-4aa1-a080-0d12dd89b0df" (UID: "d9314b72-0654-4aa1-a080-0d12dd89b0df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:52.578059 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.578020    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd" pod "d9314b72-0654-4aa1-a080-0d12dd89b0df" (UID: "d9314b72-0654-4aa1-a080-0d12dd89b0df"). InnerVolumeSpecName "kube-api-access-5jqkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:52.677134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.677101    4466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9314b72-0654-4aa1-a080-0d12dd89b0df-tls-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.677134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.677127    4466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9314b72-0654-4aa1-a080-0d12dd89b0df-dshm\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.677134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.677138    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqkd\" (UniqueName: \"kubernetes.io/projected/d9314b72-0654-4aa1-a080-0d12dd89b0df-kube-api-access-5jqkd\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:52.896715 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.896672    4466 generic.go:356] "Generic (PLEG): container finished" podID="d9314b72-0654-4aa1-a080-0d12dd89b0df" containerID="96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49" exitCode=0
Oct 31 00:48:52.896859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.896727    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"
Oct 31 00:48:52.896859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.896731    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98" event={"ID":"d9314b72-0654-4aa1-a080-0d12dd89b0df","Type":"ContainerDied","Data":"96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49"}
Oct 31 00:48:52.896947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.896843    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98" event={"ID":"d9314b72-0654-4aa1-a080-0d12dd89b0df","Type":"ContainerDied","Data":"0fdfd0ea83129f33ef117d71242e084009a212a39409c9f58d9db87c5ffa3023"}
Oct 31 00:48:52.896947 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.896879    4466 scope.go:122] "RemoveContainer" containerID="96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49"
Oct 31 00:48:52.905210 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.905191    4466 scope.go:122] "RemoveContainer" containerID="96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49"
Oct 31 00:48:52.905447 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:48:52.905432    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49\": container with ID starting with 96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49 not found: ID does not exist" containerID="96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49"
Oct 31 00:48:52.905494 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:52.905454    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49"} err="failed to get container status \"96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49\": rpc error: code = NotFound desc = could not find container \"96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49\": container with ID starting with 96ae85cbf29413322c119532a23b6ae24d8fd1aa323427bf3a13f90784dbef49 not found: ID does not exist"
Oct 31 00:48:53.069581 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:53.069551    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"]
Oct 31 00:48:53.074062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:53.074041    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d40847c7-kserve-558fb68cd5-fdm98"]
Oct 31 00:48:54.832617 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:48:54.832587    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9314b72-0654-4aa1-a080-0d12dd89b0df" path="/var/lib/kubelet/pods/d9314b72-0654-4aa1-a080-0d12dd89b0df/volumes"
Oct 31 00:49:08.316143 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:49:08.316106    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:49:27.707895 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.707860    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"]
Oct 31 00:49:27.708489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.708193    4466 cpu_manager.go:414] "containerMap: removing container" podUID="cd982356-7e6e-4d17-88c4-059bac1f55f0" containerName="main"
Oct 31 00:49:27.708489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.708203    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd982356-7e6e-4d17-88c4-059bac1f55f0" containerName="main"
Oct 31 00:49:27.708489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.708247    4466 cpu_manager.go:414] "containerMap: removing container" podUID="d9314b72-0654-4aa1-a080-0d12dd89b0df" containerName="main"
Oct 31 00:49:27.708489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.708256    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9314b72-0654-4aa1-a080-0d12dd89b0df" containerName="main"
Oct 31 00:49:27.708489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.708318    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd982356-7e6e-4d17-88c4-059bac1f55f0" containerName="main"
Oct 31 00:49:27.708489 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.708330    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9314b72-0654-4aa1-a080-0d12dd89b0df" containerName="main"
Oct 31 00:49:27.711552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.711529    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.786089 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.786053    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d28221-2948-4c1d-999e-d126f1cafe43-maas-api-tls\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.786251 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.786224    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d28221-2948-4c1d-999e-d126f1cafe43-metrics-certs\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.786325 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.786294    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6sm\" (UniqueName: \"kubernetes.io/projected/a8d28221-2948-4c1d-999e-d126f1cafe43-kube-api-access-nv6sm\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.786382 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.786363    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d28221-2948-4c1d-999e-d126f1cafe43-openshift-service-ca\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.820114 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.820084    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"]
Oct 31 00:49:27.887109 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.887077    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d28221-2948-4c1d-999e-d126f1cafe43-metrics-certs\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.887281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.887122    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sm\" (UniqueName: \"kubernetes.io/projected/a8d28221-2948-4c1d-999e-d126f1cafe43-kube-api-access-nv6sm\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.887281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.887154    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d28221-2948-4c1d-999e-d126f1cafe43-openshift-service-ca\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.887281 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.887217    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d28221-2948-4c1d-999e-d126f1cafe43-maas-api-tls\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.887910 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.887889    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d28221-2948-4c1d-999e-d126f1cafe43-openshift-service-ca\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.889522 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.889496    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d28221-2948-4c1d-999e-d126f1cafe43-metrics-certs\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.889651 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.889622    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d28221-2948-4c1d-999e-d126f1cafe43-maas-api-tls\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:27.950088 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:27.950060    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6sm\" (UniqueName: \"kubernetes.io/projected/a8d28221-2948-4c1d-999e-d126f1cafe43-kube-api-access-nv6sm\") pod \"maas-api-7879ff96d6-rthf5\" (UID: \"a8d28221-2948-4c1d-999e-d126f1cafe43\") " pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:28.024264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:28.024188    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:28.414310 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:49:28.414276    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d28221_2948_4c1d_999e_d126f1cafe43.slice/crio-80a942f821f7ce11d77e1769788f14dafd69cd1ec8fd851207255c6dbeb8e491 WatchSource:0}: Error finding container 80a942f821f7ce11d77e1769788f14dafd69cd1ec8fd851207255c6dbeb8e491: Status 404 returned error can't find the container with id 80a942f821f7ce11d77e1769788f14dafd69cd1ec8fd851207255c6dbeb8e491
Oct 31 00:49:28.415606 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:28.415583    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:49:28.417260 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:28.417239    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"]
Oct 31 00:49:29.088351 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:29.088318    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5" event={"ID":"a8d28221-2948-4c1d-999e-d126f1cafe43","Type":"ContainerStarted","Data":"80a942f821f7ce11d77e1769788f14dafd69cd1ec8fd851207255c6dbeb8e491"}
Oct 31 00:49:30.094448 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:30.094412    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5" event={"ID":"a8d28221-2948-4c1d-999e-d126f1cafe43","Type":"ContainerStarted","Data":"36cab9d2f8fd86cfca52fcb66ea8ae0820a00f33d27d76aa0d47711b57d96ea6"}
Oct 31 00:49:30.094824 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:30.094475    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:30.220779 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:30.220728    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5" podStartSLOduration=2.326879812 podStartE2EDuration="3.220714412s" totalImagesPullingTime="893.8346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:30.219815489 +0000 UTC m=+4075.510363818" watchObservedRunningTime="2026-10-31 00:49:30.220714412 +0000 UTC m=+4075.511262730"
Oct 31 00:49:36.104400 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:36.104368    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7879ff96d6-rthf5"
Oct 31 00:49:38.316096 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:49:38.316061    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:49:42.325045 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.325008    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"]
Oct 31 00:49:42.329576 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.329554    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.387709 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.387674    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"]
Oct 31 00:49:42.431126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.431096    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6sq\" (UniqueName: \"kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq\") pod \"payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.431126 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.431133    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.434818 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.434781    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8214dff9\""
Oct 31 00:49:42.434952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.434831    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8214dff9-dockercfg-7wv27\""
Oct 31 00:49:42.489031 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.488993    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"]
Oct 31 00:49:42.492474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.492456    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.532701 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.532671    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6sq\" (UniqueName: \"kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq\") pod \"payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.532934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.532715    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume\") pod \"payload-processing-e2e-ait-8214dff9-85b686f698-krx74\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") " pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.532934 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.532774    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.533100 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.533071    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l754n\" (UniqueName: \"kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n\") pod \"payload-processing-e2e-ait-8214dff9-85b686f698-krx74\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") " pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.533538 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.533502    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.538723 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.538699    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"]
Oct 31 00:49:42.596440 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.596411    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6sq\" (UniqueName: \"kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq\") pod \"payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.634063 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.634031    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l754n\" (UniqueName: \"kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n\") pod \"payload-processing-e2e-ait-8214dff9-85b686f698-krx74\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") " pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.634225 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.634081    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume\") pod \"payload-processing-e2e-ait-8214dff9-85b686f698-krx74\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") " pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.634790 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.634772    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume\") pod \"payload-processing-e2e-ait-8214dff9-85b686f698-krx74\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") " pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.642983 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.642957    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:42.717566 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.717496    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l754n\" (UniqueName: \"kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n\") pod \"payload-processing-e2e-ait-8214dff9-85b686f698-krx74\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") " pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:42.810591 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:42.810556    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:43.010842 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:49:43.010803    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170dad73_29e6_40fe_bf93_dd6773e421ea.slice/crio-130ee2e394b7b8e82876b8271150402504c7043380f4168871113399e4d442f6 WatchSource:0}: Error finding container 130ee2e394b7b8e82876b8271150402504c7043380f4168871113399e4d442f6: Status 404 returned error can't find the container with id 130ee2e394b7b8e82876b8271150402504c7043380f4168871113399e4d442f6
Oct 31 00:49:43.013515 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.013491    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"]
Oct 31 00:49:43.154403 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.154323    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h" event={"ID":"170dad73-29e6-40fe-bf93-dd6773e421ea","Type":"ContainerStarted","Data":"ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df"}
Oct 31 00:49:43.154403 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.154372    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h" event={"ID":"170dad73-29e6-40fe-bf93-dd6773e421ea","Type":"ContainerStarted","Data":"130ee2e394b7b8e82876b8271150402504c7043380f4168871113399e4d442f6"}
Oct 31 00:49:43.154623 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.154408    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:43.182528 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:49:43.182501    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71728d2b_392e_498d_8d2b_69adccb8ba71.slice/crio-570f48fc5b5c2187143ea63ce7891b6168e3aba9a84bdb26932e8069f08e9be2 WatchSource:0}: Error finding container 570f48fc5b5c2187143ea63ce7891b6168e3aba9a84bdb26932e8069f08e9be2: Status 404 returned error can't find the container with id 570f48fc5b5c2187143ea63ce7891b6168e3aba9a84bdb26932e8069f08e9be2
Oct 31 00:49:43.185937 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.185912    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"]
Oct 31 00:49:43.275170 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.275121    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h" podStartSLOduration=1.275107728 podStartE2EDuration="1.275107728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:43.274148163 +0000 UTC m=+4088.564696481" watchObservedRunningTime="2026-10-31 00:49:43.275107728 +0000 UTC m=+4088.565656040"
Oct 31 00:49:43.621697 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.621666    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"]
Oct 31 00:49:43.625101 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.625085    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.643454 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.643432    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.643582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.643490    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.643651 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.643598    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.643651 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.643622    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl44r\" (UniqueName: \"kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.682111 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.682083    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"]
Oct 31 00:49:43.732817 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.732791    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8214dff9-serving-cert\""
Oct 31 00:49:43.744832 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.744806    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.744972 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.744849    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.744972 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.744880    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl44r\" (UniqueName: \"kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.744972 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.744928    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.745088 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:49:43.745060    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-serving-cert: secret "maas-api-e2e-ait-8214dff9-serving-cert" not found
Oct 31 00:49:43.745133 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:49:43.745124    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls podName:bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab nodeName:}" failed. No retries permitted until 2026-10-31 00:49:44.245105758 +0000 UTC m=+4089.535654075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls") pod "maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" (UID: "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab") : secret "maas-api-e2e-ait-8214dff9-serving-cert" not found
Oct 31 00:49:43.745459 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.745433    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.747426 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.747402    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:43.808767 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:43.808736    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl44r\" (UniqueName: \"kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:44.159733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.159694    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74" event={"ID":"71728d2b-392e-498d-8d2b-69adccb8ba71","Type":"ContainerStarted","Data":"b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194"}
Oct 31 00:49:44.159922 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.159755    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74" event={"ID":"71728d2b-392e-498d-8d2b-69adccb8ba71","Type":"ContainerStarted","Data":"570f48fc5b5c2187143ea63ce7891b6168e3aba9a84bdb26932e8069f08e9be2"}
Oct 31 00:49:44.159922 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.159804    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:49:44.249754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.249718    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:44.251988 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.251964    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\") pod \"maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:44.281223 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.281173    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74" podStartSLOduration=2.281159167 podStartE2EDuration="2.281159167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:44.279804463 +0000 UTC m=+4089.570352785" watchObservedRunningTime="2026-10-31 00:49:44.281159167 +0000 UTC m=+4089.571707484"
Oct 31 00:49:44.537741 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.537658    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:44.905626 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:49:44.905587    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9638c4_48c4_4bd3_b69a_29cbc1aa67ab.slice/crio-af149355467d23f67b602a445b8c77bf2668842ad7580989fefa0f9bbe777a1f WatchSource:0}: Error finding container af149355467d23f67b602a445b8c77bf2668842ad7580989fefa0f9bbe777a1f: Status 404 returned error can't find the container with id af149355467d23f67b602a445b8c77bf2668842ad7580989fefa0f9bbe777a1f
Oct 31 00:49:44.908849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:44.908825    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"]
Oct 31 00:49:45.165715 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:45.165621    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" event={"ID":"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab","Type":"ContainerStarted","Data":"af149355467d23f67b602a445b8c77bf2668842ad7580989fefa0f9bbe777a1f"}
Oct 31 00:49:45.375474 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:45.375440    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"]
Oct 31 00:49:45.375691 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:45.375656    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" podUID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" containerName="maas-api" containerID="cri-o://b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77" gracePeriod=30
Oct 31 00:49:46.171733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.171653    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" event={"ID":"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab","Type":"ContainerStarted","Data":"31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b"}
Oct 31 00:49:46.171733 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.171714    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:46.209810 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.209778    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" podUID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.23:8443/health\": dial tcp 10.133.0.23:8443: connect: connection refused"
Oct 31 00:49:46.295330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.295277    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" podStartSLOduration=2.320296529 podStartE2EDuration="3.295263372s" totalImagesPullingTime="974.966843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:46.293470043 +0000 UTC m=+4091.584018363" watchObservedRunningTime="2026-10-31 00:49:46.295263372 +0000 UTC m=+4091.585811732"
Oct 31 00:49:46.625960 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.625936    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 31 00:49:46.672823 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.672791    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca\") pod \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") "
Oct 31 00:49:46.672985 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.672906    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls\") pod \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") "
Oct 31 00:49:46.673046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.672987    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs\") pod \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") "
Oct 31 00:49:46.673087 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.673061    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q\" (UniqueName: \"kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q\") pod \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\" (UID: \"653ca4d3-0a74-45c5-b15a-f5e2076fa656\") "
Oct 31 00:49:46.673144 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.673118    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca" pod "653ca4d3-0a74-45c5-b15a-f5e2076fa656" (UID: "653ca4d3-0a74-45c5-b15a-f5e2076fa656"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:46.673334 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.673319    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653ca4d3-0a74-45c5-b15a-f5e2076fa656-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.675046 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.675019    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs" pod "653ca4d3-0a74-45c5-b15a-f5e2076fa656" (UID: "653ca4d3-0a74-45c5-b15a-f5e2076fa656"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:46.675300 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.675088    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q" pod "653ca4d3-0a74-45c5-b15a-f5e2076fa656" (UID: "653ca4d3-0a74-45c5-b15a-f5e2076fa656"). InnerVolumeSpecName "kube-api-access-jcd9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:46.675300 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.675108    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls" pod "653ca4d3-0a74-45c5-b15a-f5e2076fa656" (UID: "653ca4d3-0a74-45c5-b15a-f5e2076fa656"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:46.774412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.774335    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.774412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.774375    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcd9q\" (UniqueName: \"kubernetes.io/projected/653ca4d3-0a74-45c5-b15a-f5e2076fa656-kube-api-access-jcd9q\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.774412 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:46.774392    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653ca4d3-0a74-45c5-b15a-f5e2076fa656-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:47.177883 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.177850    4466 generic.go:356] "Generic (PLEG): container finished" podID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" containerID="b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77" exitCode=0
Oct 31 00:49:47.178358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.177909    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"
Oct 31 00:49:47.178358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.177937    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" event={"ID":"653ca4d3-0a74-45c5-b15a-f5e2076fa656","Type":"ContainerDied","Data":"b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77"}
Oct 31 00:49:47.178358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.178003    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r" event={"ID":"653ca4d3-0a74-45c5-b15a-f5e2076fa656","Type":"ContainerDied","Data":"fc02ad1d0270c58b997b967d41c0cce45461439e8baee4a0f319b1391f60ab6a"}
Oct 31 00:49:47.178358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.178035    4466 scope.go:122] "RemoveContainer" containerID="b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77"
Oct 31 00:49:47.187129 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.187077    4466 scope.go:122] "RemoveContainer" containerID="b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77"
Oct 31 00:49:47.189168 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:49:47.189140    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77\": container with ID starting with b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77 not found: ID does not exist" containerID="b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77"
Oct 31 00:49:47.189264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.189175    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77"} err="failed to get container status \"b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77\": rpc error: code = NotFound desc = could not find container \"b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77\": container with ID starting with b503d8f1ca0827dfaa0d2eff20a74eab6ebea8e589866409efa4860080f6db77 not found: ID does not exist"
Oct 31 00:49:47.350056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.350024    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"]
Oct 31 00:49:47.354586 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:47.354561    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56bf677d9d-kn27r"]
Oct 31 00:49:48.825356 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:48.825327    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" path="/var/lib/kubelet/pods/653ca4d3-0a74-45c5-b15a-f5e2076fa656/volumes"
Oct 31 00:49:52.183014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:52.182983    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:49:54.162509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:54.162475    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:49:55.168333 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:49:55.168309    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:50:08.315795 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:08.315753    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:50:26.640809 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:26.640777    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"]
Oct 31 00:50:26.739799 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:26.739765    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"]
Oct 31 00:50:31.641342 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:31.641277    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74" podUID="71728d2b-392e-498d-8d2b-69adccb8ba71" containerName="payload-processing" containerID="cri-o://b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194" gracePeriod=25
Oct 31 00:50:31.740372 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:31.740321    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h" podUID="170dad73-29e6-40fe-bf93-dd6773e421ea" containerName="payload-pre-processing" containerID="cri-o://ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df" gracePeriod=25
Oct 31 00:50:31.900066 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:31.900008    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:50:32.006138 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.006117    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:50:32.008311 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.008293    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume\") pod \"71728d2b-392e-498d-8d2b-69adccb8ba71\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") "
Oct 31 00:50:32.008399 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.008328    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n\" (UniqueName: \"kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n\") pod \"71728d2b-392e-498d-8d2b-69adccb8ba71\" (UID: \"71728d2b-392e-498d-8d2b-69adccb8ba71\") "
Oct 31 00:50:32.008664 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.008626    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume" pod "71728d2b-392e-498d-8d2b-69adccb8ba71" (UID: "71728d2b-392e-498d-8d2b-69adccb8ba71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:32.010192 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.010174    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n" pod "71728d2b-392e-498d-8d2b-69adccb8ba71" (UID: "71728d2b-392e-498d-8d2b-69adccb8ba71"). InnerVolumeSpecName "kube-api-access-l754n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:32.109142 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.109116    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq\" (UniqueName: \"kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq\") pod \"170dad73-29e6-40fe-bf93-dd6773e421ea\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") "
Oct 31 00:50:32.109278 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.109240    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume\") pod \"170dad73-29e6-40fe-bf93-dd6773e421ea\" (UID: \"170dad73-29e6-40fe-bf93-dd6773e421ea\") "
Oct 31 00:50:32.109552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.109533    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71728d2b-392e-498d-8d2b-69adccb8ba71-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:32.109614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.109547    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume" pod "170dad73-29e6-40fe-bf93-dd6773e421ea" (UID: "170dad73-29e6-40fe-bf93-dd6773e421ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:32.109614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.109552    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l754n\" (UniqueName: \"kubernetes.io/projected/71728d2b-392e-498d-8d2b-69adccb8ba71-kube-api-access-l754n\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:32.111158 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.111142    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq" pod "170dad73-29e6-40fe-bf93-dd6773e421ea" (UID: "170dad73-29e6-40fe-bf93-dd6773e421ea"). InnerVolumeSpecName "kube-api-access-dq6sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:32.210521 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.210447    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170dad73-29e6-40fe-bf93-dd6773e421ea-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:32.210521 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.210473    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6sq\" (UniqueName: \"kubernetes.io/projected/170dad73-29e6-40fe-bf93-dd6773e421ea-kube-api-access-dq6sq\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:32.384338 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.384304    4466 generic.go:356] "Generic (PLEG): container finished" podID="170dad73-29e6-40fe-bf93-dd6773e421ea" containerID="ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df" exitCode=0
Oct 31 00:50:32.384507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.384346    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"
Oct 31 00:50:32.384507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.384389    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h" event={"ID":"170dad73-29e6-40fe-bf93-dd6773e421ea","Type":"ContainerDied","Data":"ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df"}
Oct 31 00:50:32.384507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.384442    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h" event={"ID":"170dad73-29e6-40fe-bf93-dd6773e421ea","Type":"ContainerDied","Data":"130ee2e394b7b8e82876b8271150402504c7043380f4168871113399e4d442f6"}
Oct 31 00:50:32.384507 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.384470    4466 scope.go:122] "RemoveContainer" containerID="ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df"
Oct 31 00:50:32.385654 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.385614    4466 generic.go:356] "Generic (PLEG): container finished" podID="71728d2b-392e-498d-8d2b-69adccb8ba71" containerID="b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194" exitCode=0
Oct 31 00:50:32.385754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.385670    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74" event={"ID":"71728d2b-392e-498d-8d2b-69adccb8ba71","Type":"ContainerDied","Data":"b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194"}
Oct 31 00:50:32.385754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.385678    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"
Oct 31 00:50:32.385754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.385709    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74" event={"ID":"71728d2b-392e-498d-8d2b-69adccb8ba71","Type":"ContainerDied","Data":"570f48fc5b5c2187143ea63ce7891b6168e3aba9a84bdb26932e8069f08e9be2"}
Oct 31 00:50:32.393966 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.393947    4466 scope.go:122] "RemoveContainer" containerID="ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df"
Oct 31 00:50:32.394214 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:32.394197    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df\": container with ID starting with ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df not found: ID does not exist" containerID="ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df"
Oct 31 00:50:32.394285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.394229    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df"} err="failed to get container status \"ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df\": rpc error: code = NotFound desc = could not find container \"ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df\": container with ID starting with ac2d48d18429dd8e9b21fc92df4fc94838ba7d5045cfa6d5c5b89ab0adf875df not found: ID does not exist"
Oct 31 00:50:32.394285 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.394245    4466 scope.go:122] "RemoveContainer" containerID="b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194"
Oct 31 00:50:32.401614 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.401596    4466 scope.go:122] "RemoveContainer" containerID="b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194"
Oct 31 00:50:32.401971 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:32.401947    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194\": container with ID starting with b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194 not found: ID does not exist" containerID="b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194"
Oct 31 00:50:32.402014 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.401972    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194"} err="failed to get container status \"b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194\": rpc error: code = NotFound desc = could not find container \"b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194\": container with ID starting with b169931b86f766398dffc182184ba044cb0f839fc9d507e2a956157dee0fb194 not found: ID does not exist"
Oct 31 00:50:32.559765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.559692    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"]
Oct 31 00:50:32.611998 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.611968    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8214dff9-85b686f698-krx74"]
Oct 31 00:50:32.725698 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.725658    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"]
Oct 31 00:50:32.730238 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.730204    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8214dff9-854cb688f9-z4k5h"]
Oct 31 00:50:32.826187 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.826127    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170dad73-29e6-40fe-bf93-dd6773e421ea" path="/var/lib/kubelet/pods/170dad73-29e6-40fe-bf93-dd6773e421ea/volumes"
Oct 31 00:50:32.826479 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:32.826466    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71728d2b-392e-498d-8d2b-69adccb8ba71" path="/var/lib/kubelet/pods/71728d2b-392e-498d-8d2b-69adccb8ba71/volumes"
Oct 31 00:50:34.486296 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.486263    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"]
Oct 31 00:50:34.486776 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.486512    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" podUID="bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" containerName="maas-api" containerID="cri-o://31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b" gracePeriod=30
Oct 31 00:50:34.739758 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.739703    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:50:34.835171 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.835145    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca\") pod \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") "
Oct 31 00:50:34.835312 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.835196    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r\" (UniqueName: \"kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r\") pod \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") "
Oct 31 00:50:34.835312 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.835274    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs\") pod \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") "
Oct 31 00:50:34.835429 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.835318    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\") pod \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\" (UID: \"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab\") "
Oct 31 00:50:34.835512 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.835485    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca" pod "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" (UID: "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:34.835656 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.835625    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:34.837100 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.837072    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r" pod "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" (UID: "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab"). InnerVolumeSpecName "kube-api-access-wl44r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:34.837344 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.837319    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls" pod "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" (UID: "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:50:34.837423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.837336    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs" pod "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" (UID: "bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:50:34.936466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.936439    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl44r\" (UniqueName: \"kubernetes.io/projected/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-kube-api-access-wl44r\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:34.936466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.936462    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:34.936466 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:34.936471    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:35.402038 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.402006    4466 generic.go:356] "Generic (PLEG): container finished" podID="bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" containerID="31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b" exitCode=0
Oct 31 00:50:35.402216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.402049    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"
Oct 31 00:50:35.402216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.402068    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" event={"ID":"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab","Type":"ContainerDied","Data":"31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b"}
Oct 31 00:50:35.402216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.402105    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf" event={"ID":"bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab","Type":"ContainerDied","Data":"af149355467d23f67b602a445b8c77bf2668842ad7580989fefa0f9bbe777a1f"}
Oct 31 00:50:35.402216 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.402126    4466 scope.go:122] "RemoveContainer" containerID="31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b"
Oct 31 00:50:35.413043 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.413024    4466 scope.go:122] "RemoveContainer" containerID="31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b"
Oct 31 00:50:35.413330 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:35.413313    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b\": container with ID starting with 31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b not found: ID does not exist" containerID="31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b"
Oct 31 00:50:35.413383 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.413340    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b"} err="failed to get container status \"31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b\": rpc error: code = NotFound desc = could not find container \"31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b\": container with ID starting with 31e561c64d53a91964ab16918894c264328623a1434684b3fdf846d19d284d5b not found: ID does not exist"
Oct 31 00:50:35.573790 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.573759    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"]
Oct 31 00:50:35.577156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:35.577131    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8214dff9-67bb47b8d5-rlrsf"]
Oct 31 00:50:36.825865 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:36.825837    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" path="/var/lib/kubelet/pods/bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab/volumes"
Oct 31 00:50:38.315003 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:38.314971    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:50:43.139751 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.139716    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"]
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140082    4466 cpu_manager.go:414] "containerMap: removing container" podUID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" containerName="maas-api"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140093    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" containerName="maas-api"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140121    4466 cpu_manager.go:414] "containerMap: removing container" podUID="170dad73-29e6-40fe-bf93-dd6773e421ea" containerName="payload-pre-processing"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140126    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170dad73-29e6-40fe-bf93-dd6773e421ea" containerName="payload-pre-processing"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140138    4466 cpu_manager.go:414] "containerMap: removing container" podUID="71728d2b-392e-498d-8d2b-69adccb8ba71" containerName="payload-processing"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140143    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71728d2b-392e-498d-8d2b-69adccb8ba71" containerName="payload-processing"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140150    4466 cpu_manager.go:414] "containerMap: removing container" podUID="bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" containerName="maas-api"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140154    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" containerName="maas-api"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140228    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="71728d2b-392e-498d-8d2b-69adccb8ba71" containerName="payload-processing"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140234    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9638c4-48c4-4bd3-b69a-29cbc1aa67ab" containerName="maas-api"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140241    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="653ca4d3-0a74-45c5-b15a-f5e2076fa656" containerName="maas-api"
Oct 31 00:50:43.140252 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.140250    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="170dad73-29e6-40fe-bf93-dd6773e421ea" containerName="payload-pre-processing"
Oct 31 00:50:43.143210 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.143193    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210227    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210268    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210498 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210286    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210498 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210426    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpt6k\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210504    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210582 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210541    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210704 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210686    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210754 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210732    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.210843 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.210827    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.234873 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.234847    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-a3c14c-openshift-default-dockercfg-r2nf8\""
Oct 31 00:50:43.280123 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.280091    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"]
Oct 31 00:50:43.312184 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312160    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312196    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312227    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312253    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312275    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312292    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312337 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312315    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpt6k\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312372    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312398    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312662    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312728 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312711    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312778    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.312952 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.312839    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.313073 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.313054    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.314731 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.314707    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.314825 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.314790    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.377485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.377448    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpt6k\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.383678 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.383646    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token\") pod \"e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") " pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.456028 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.455945    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:43.878801 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:50:43.878774    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09b24d3_dae8_4d07_a741_a0aedc6f6bf0.slice/crio-d249bf237d24d116bf47ac8395e825aae90a5e44c983aa59f686597e71daa65a WatchSource:0}: Error finding container d249bf237d24d116bf47ac8395e825aae90a5e44c983aa59f686597e71daa65a: Status 404 returned error can't find the container with id d249bf237d24d116bf47ac8395e825aae90a5e44c983aa59f686597e71daa65a
Oct 31 00:50:43.881009 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.880978    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:50:43.881112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.881041    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:50:43.881112 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.881071    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:50:43.882073 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:43.882051    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"]
Oct 31 00:50:44.443615 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:44.443579    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" event={"ID":"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0","Type":"ContainerStarted","Data":"9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"}
Oct 31 00:50:44.444004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:44.443657    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" event={"ID":"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0","Type":"ContainerStarted","Data":"d249bf237d24d116bf47ac8395e825aae90a5e44c983aa59f686597e71daa65a"}
Oct 31 00:50:44.457020 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:44.456999    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:44.461707 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:44.461686    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:44.571332 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:44.571287    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podStartSLOduration=1.5712732969999998 podStartE2EDuration="1.571273297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:44.569216078 +0000 UTC m=+4149.859764397" watchObservedRunningTime="2026-10-31 00:50:44.571273297 +0000 UTC m=+4149.861821614"
Oct 31 00:50:45.448689 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:45.448659    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:45.449673 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:45.449653    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:50:50.906485 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:50.906450    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"]
Oct 31 00:50:50.909950 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:50.909932    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:50.986163 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:50.986137    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"]
Oct 31 00:50:51.036056 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.036026    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-a3c14c\""
Oct 31 00:50:51.036229 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.036082    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-a3c14c-dockercfg-shncd\""
Oct 31 00:50:51.109737 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.109708    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.109886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.109762    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d\") pod \"payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.210501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.210433    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.210501 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.210486    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d\") pod \"payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.211200 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.211179    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.273846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.273818    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d\") pod \"payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.522819 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.522741    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:51.891133 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:50:51.891105    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebba3459_19c3_4bf2_9e18_2db0b5a29327.slice/crio-ce6a47f4880736d9f47ec76d6c42f8c40aa1d11a6335225ef0ba2a338cbfcb56 WatchSource:0}: Error finding container ce6a47f4880736d9f47ec76d6c42f8c40aa1d11a6335225ef0ba2a338cbfcb56: Status 404 returned error can't find the container with id ce6a47f4880736d9f47ec76d6c42f8c40aa1d11a6335225ef0ba2a338cbfcb56
Oct 31 00:50:51.893407 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:51.893372    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"]
Oct 31 00:50:52.480040 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:52.479999    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc" event={"ID":"ebba3459-19c3-4bf2-9e18-2db0b5a29327","Type":"ContainerStarted","Data":"238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a"}
Oct 31 00:50:52.480541 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:52.480057    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc" event={"ID":"ebba3459-19c3-4bf2-9e18-2db0b5a29327","Type":"ContainerStarted","Data":"ce6a47f4880736d9f47ec76d6c42f8c40aa1d11a6335225ef0ba2a338cbfcb56"}
Oct 31 00:50:52.480541 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:52.480170    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:50:52.601127 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:52.601074    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc" podStartSLOduration=2.601061188 podStartE2EDuration="2.601061188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:52.600775462 +0000 UTC m=+4157.891323779" watchObservedRunningTime="2026-10-31 00:50:52.601061188 +0000 UTC m=+4157.891609502"
Oct 31 00:50:53.702765 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.702727    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"]
Oct 31 00:50:53.707967 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.707928    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.732672 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.732618    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.732813 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.732727    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.732813 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.732788    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dr7p\" (UniqueName: \"kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.732933 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.732811    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.762837 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.762816    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-a3c14c-serving-cert\""
Oct 31 00:50:53.817149 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.817121    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"]
Oct 31 00:50:53.834371 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.834337    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.834599 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.834417    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.834599 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.834482    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dr7p\" (UniqueName: \"kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.834599 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:53.834506    4466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-serving-cert: secret "maas-api-e2e-authcfg-http-a3c14c-serving-cert" not found
Oct 31 00:50:53.834599 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:50:53.834584    4466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls podName:ab1f6566-933e-4e88-aa17-9cbfe8e722b3 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:54.33456412 +0000 UTC m=+4159.625112432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls") pod "maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" (UID: "ab1f6566-933e-4e88-aa17-9cbfe8e722b3") : secret "maas-api-e2e-authcfg-http-a3c14c-serving-cert" not found
Oct 31 00:50:53.834940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.834511    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.835774 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.835748    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.837547 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.837516    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:53.899756 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:53.899732    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dr7p\" (UniqueName: \"kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:54.339739 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:54.339707    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:54.342123 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:54.342101    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:54.620715 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:54.620621    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:54.989946 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:50:54.989915    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1f6566_933e_4e88_aa17_9cbfe8e722b3.slice/crio-0ef9b5811c019fd65b5d83690308e0104ed716ec4275f1c2b739fc318c9b8ce3 WatchSource:0}: Error finding container 0ef9b5811c019fd65b5d83690308e0104ed716ec4275f1c2b739fc318c9b8ce3: Status 404 returned error can't find the container with id 0ef9b5811c019fd65b5d83690308e0104ed716ec4275f1c2b739fc318c9b8ce3
Oct 31 00:50:54.992892 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:54.992868    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"]
Oct 31 00:50:55.493699 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:55.493662    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" event={"ID":"ab1f6566-933e-4e88-aa17-9cbfe8e722b3","Type":"ContainerStarted","Data":"0ef9b5811c019fd65b5d83690308e0104ed716ec4275f1c2b739fc318c9b8ce3"}
Oct 31 00:50:56.499578 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:56.499538    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" event={"ID":"ab1f6566-933e-4e88-aa17-9cbfe8e722b3","Type":"ContainerStarted","Data":"190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e"}
Oct 31 00:50:56.500071 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:56.499663    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:50:56.623110 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:50:56.623060    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" podStartSLOduration=2.7250451079999998 podStartE2EDuration="3.623046109s" totalImagesPullingTime="898.001001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:56.621696193 +0000 UTC m=+4161.912244511" watchObservedRunningTime="2026-10-31 00:50:56.623046109 +0000 UTC m=+4161.913594426"
Oct 31 00:51:02.509363 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:02.509335    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:51:03.487819 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:03.487788    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:51:08.316853 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:51:08.316812    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:51:10.585405 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:10.585368    4466 generic.go:356] "Generic (PLEG): container finished" podID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerID="9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639" exitCode=0
Oct 31 00:51:10.585880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:10.585442    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" event={"ID":"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0","Type":"ContainerDied","Data":"9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"}
Oct 31 00:51:10.585880 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:10.585814    4466 scope.go:122] "RemoveContainer" containerID="9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"
Oct 31 00:51:10.586977 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:10.586948    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:51:10.587062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:10.587014    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:51:10.587062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:10.587044    4466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 00:51:11.591886 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:11.591854    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" event={"ID":"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0","Type":"ContainerStarted","Data":"cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2"}
Oct 31 00:51:12.457132 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:12.457096    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:51:12.458026 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:12.458003    4466 patch_prober.go:29] interesting pod/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:51:12.458141 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:12.458048    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 00:51:13.456556 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:13.456521    4466 patch_prober.go:29] interesting pod/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:51:13.456964 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:13.456583    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 00:51:14.457096 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:14.457065    4466 patch_prober.go:29] interesting pod/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:51:14.457555 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:14.457114    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 00:51:15.448759 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:15.448723    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:51:15.456484 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:15.456453    4466 patch_prober.go:29] interesting pod/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:51:15.456605 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:15.456513    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 00:51:16.457224 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:16.457124    4466 patch_prober.go:29] interesting pod/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:51:16.457224 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:16.457179    4466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 00:51:17.461354 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:17.461325    4466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:51:17.462148 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:51:17.462128    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:51:38.316749 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:51:38.316708    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:52:08.315088 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:52:08.315055    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:52:38.315486 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:52:38.315446    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:52:44.064022 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:44.063985    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"]
Oct 31 00:52:49.065648 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.065520    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc" podUID="ebba3459-19c3-4bf2-9e18-2db0b5a29327" containerName="payload-pre-processing" containerID="cri-o://238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a" gracePeriod=25
Oct 31 00:52:49.312943 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.312918    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:52:49.356768 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.356689    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume\") pod \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") "
Oct 31 00:52:49.356900 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.356852    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d\") pod \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\" (UID: \"ebba3459-19c3-4bf2-9e18-2db0b5a29327\") "
Oct 31 00:52:49.356969 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.356938    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume" pod "ebba3459-19c3-4bf2-9e18-2db0b5a29327" (UID: "ebba3459-19c3-4bf2-9e18-2db0b5a29327"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:49.357115 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.357099    4466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebba3459-19c3-4bf2-9e18-2db0b5a29327-plugins-config-volume\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:49.358889 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.358867    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d" pod "ebba3459-19c3-4bf2-9e18-2db0b5a29327" (UID: "ebba3459-19c3-4bf2-9e18-2db0b5a29327"). InnerVolumeSpecName "kube-api-access-cqm9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:49.458106 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:49.458080    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/ebba3459-19c3-4bf2-9e18-2db0b5a29327-kube-api-access-cqm9d\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:50.020241 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.020204    4466 generic.go:356] "Generic (PLEG): container finished" podID="ebba3459-19c3-4bf2-9e18-2db0b5a29327" containerID="238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a" exitCode=0
Oct 31 00:52:50.020241 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.020252    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"
Oct 31 00:52:50.020509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.020297    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc" event={"ID":"ebba3459-19c3-4bf2-9e18-2db0b5a29327","Type":"ContainerDied","Data":"238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a"}
Oct 31 00:52:50.020509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.020360    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc" event={"ID":"ebba3459-19c3-4bf2-9e18-2db0b5a29327","Type":"ContainerDied","Data":"ce6a47f4880736d9f47ec76d6c42f8c40aa1d11a6335225ef0ba2a338cbfcb56"}
Oct 31 00:52:50.020509 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.020392    4466 scope.go:122] "RemoveContainer" containerID="238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a"
Oct 31 00:52:50.029428 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.029412    4466 scope.go:122] "RemoveContainer" containerID="238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a"
Oct 31 00:52:50.029721 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:52:50.029697    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a\": container with ID starting with 238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a not found: ID does not exist" containerID="238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a"
Oct 31 00:52:50.029816 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.029734    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a"} err="failed to get container status \"238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a\": rpc error: code = NotFound desc = could not find container \"238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a\": container with ID starting with 238970f78f8bd787df7dae51a36aeae890218279746d71004cbbd980a302cf2a not found: ID does not exist"
Oct 31 00:52:50.190846 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.190818    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"]
Oct 31 00:52:50.248825 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.244736    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-a3c14c-775858b75-gthbc"]
Oct 31 00:52:50.825839 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:50.825806    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebba3459-19c3-4bf2-9e18-2db0b5a29327" path="/var/lib/kubelet/pods/ebba3459-19c3-4bf2-9e18-2db0b5a29327/volumes"
Oct 31 00:52:54.964139 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:54.964096    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"]
Oct 31 00:52:54.964626 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:54.964329    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" podUID="ab1f6566-933e-4e88-aa17-9cbfe8e722b3" containerName="maas-api" containerID="cri-o://190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e" gracePeriod=30
Oct 31 00:52:55.219627 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.219569    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:52:55.309901 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.309864    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs\") pod \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") "
Oct 31 00:52:55.310059 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.309950    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\") pod \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") "
Oct 31 00:52:55.310059 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.309983    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p\" (UniqueName: \"kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p\") pod \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") "
Oct 31 00:52:55.310059 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.310006    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca\") pod \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\" (UID: \"ab1f6566-933e-4e88-aa17-9cbfe8e722b3\") "
Oct 31 00:52:55.310440 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.310417    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca" pod "ab1f6566-933e-4e88-aa17-9cbfe8e722b3" (UID: "ab1f6566-933e-4e88-aa17-9cbfe8e722b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:55.312094 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.312064    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p" pod "ab1f6566-933e-4e88-aa17-9cbfe8e722b3" (UID: "ab1f6566-933e-4e88-aa17-9cbfe8e722b3"). InnerVolumeSpecName "kube-api-access-2dr7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:55.312206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.312102    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls" pod "ab1f6566-933e-4e88-aa17-9cbfe8e722b3" (UID: "ab1f6566-933e-4e88-aa17-9cbfe8e722b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:55.312206 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.312142    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs" pod "ab1f6566-933e-4e88-aa17-9cbfe8e722b3" (UID: "ab1f6566-933e-4e88-aa17-9cbfe8e722b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:55.411286 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.411251    4466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-metrics-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:55.411286 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.411281    4466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-maas-api-tls\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:55.411286 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.411291    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dr7p\" (UniqueName: \"kubernetes.io/projected/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-kube-api-access-2dr7p\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:55.411528 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:55.411301    4466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1f6566-933e-4e88-aa17-9cbfe8e722b3-openshift-service-ca\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:56.050376 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.050340    4466 generic.go:356] "Generic (PLEG): container finished" podID="ab1f6566-933e-4e88-aa17-9cbfe8e722b3" containerID="190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e" exitCode=0
Oct 31 00:52:56.050798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.050385    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"
Oct 31 00:52:56.050798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.050422    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" event={"ID":"ab1f6566-933e-4e88-aa17-9cbfe8e722b3","Type":"ContainerDied","Data":"190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e"}
Oct 31 00:52:56.050798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.050479    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml" event={"ID":"ab1f6566-933e-4e88-aa17-9cbfe8e722b3","Type":"ContainerDied","Data":"0ef9b5811c019fd65b5d83690308e0104ed716ec4275f1c2b739fc318c9b8ce3"}
Oct 31 00:52:56.050798 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.050507    4466 scope.go:122] "RemoveContainer" containerID="190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e"
Oct 31 00:52:56.059830 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.059809    4466 scope.go:122] "RemoveContainer" containerID="190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e"
Oct 31 00:52:56.060098 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:52:56.060081    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e\": container with ID starting with 190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e not found: ID does not exist" containerID="190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e"
Oct 31 00:52:56.060156 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.060108    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e"} err="failed to get container status \"190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e\": rpc error: code = NotFound desc = could not find container \"190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e\": container with ID starting with 190dd5a6766f16437d5289612289179e48c9b9479f261c1d23c9e7a9f142c26e not found: ID does not exist"
Oct 31 00:52:56.222607 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.222573    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"]
Oct 31 00:52:56.274789 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.274761    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-a3c14c-584bfdb945-4z8ml"]
Oct 31 00:52:56.825423 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:52:56.825393    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1f6566-933e-4e88-aa17-9cbfe8e722b3" path="/var/lib/kubelet/pods/ab1f6566-933e-4e88-aa17-9cbfe8e722b3/volumes"
Oct 31 00:53:04.424444 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:04.424411    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"]
Oct 31 00:53:04.424868 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:04.424664    4466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy" containerID="cri-o://cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2" gracePeriod=30
Oct 31 00:53:08.315407 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:53:08.315373    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:53:09.676997 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.676967    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:53:09.739461 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739394    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739461 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739447    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739474    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739504    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739550    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739690    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739814    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.739859 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739848    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:09.740024 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739854    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.740024 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739908    4466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token\") pod \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\" (UID: \"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0\") "
Oct 31 00:53:09.740024 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.739928    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:09.740024 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740018    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:09.740228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740127    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:09.740228 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740185    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:09.740358 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740331    4466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istiod-ca-cert\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.740421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740356    4466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-credential-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.740421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740371    4466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-socket\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.740421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740385    4466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-data\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.740421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.740398    4466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-workload-certs\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.742004 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.741982    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:09.742113 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.742031    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:09.742113 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.742063    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "kube-api-access-wpt6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:09.742113 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.742078    4466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo" pod "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" (UID: "c09b24d3-dae8-4d07-a741-a0aedc6f6bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:53:09.841686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.841662    4466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-token\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.841686 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.841684    4466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-podinfo\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.841850 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.841694    4466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-istio-envoy\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:09.841850 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:09.841703    4466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpt6k\" (UniqueName: \"kubernetes.io/projected/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0-kube-api-access-wpt6k\") on node \"ip-10-0-131-123.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:10.114110 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.114075    4466 generic.go:356] "Generic (PLEG): container finished" podID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerID="cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2" exitCode=0
Oct 31 00:53:10.114304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.114119    4466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"
Oct 31 00:53:10.114304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.114118    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" event={"ID":"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0","Type":"ContainerDied","Data":"cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2"}
Oct 31 00:53:10.114304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.114243    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4" event={"ID":"c09b24d3-dae8-4d07-a741-a0aedc6f6bf0","Type":"ContainerDied","Data":"d249bf237d24d116bf47ac8395e825aae90a5e44c983aa59f686597e71daa65a"}
Oct 31 00:53:10.114304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.114266    4466 scope.go:122] "RemoveContainer" containerID="cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2"
Oct 31 00:53:10.123849 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.123826    4466 scope.go:122] "RemoveContainer" containerID="9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"
Oct 31 00:53:10.131002 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.130984    4466 scope.go:122] "RemoveContainer" containerID="cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2"
Oct 31 00:53:10.131226 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:53:10.131209    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2\": container with ID starting with cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2 not found: ID does not exist" containerID="cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2"
Oct 31 00:53:10.131272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.131233    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2"} err="failed to get container status \"cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2\": rpc error: code = NotFound desc = could not find container \"cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2\": container with ID starting with cad61bf54a3647319384030a2b569efabe72e7f8b71887e2c0ecf01011e5c6b2 not found: ID does not exist"
Oct 31 00:53:10.131272 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.131248    4466 scope.go:122] "RemoveContainer" containerID="9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"
Oct 31 00:53:10.131441 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:53:10.131426    4466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639\": container with ID starting with 9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639 not found: ID does not exist" containerID="9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"
Oct 31 00:53:10.131480 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.131445    4466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639"} err="failed to get container status \"9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639\": rpc error: code = NotFound desc = could not find container \"9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639\": container with ID starting with 9c994e1309d097960cca1aef95d714f4b2c2b61067a8a7eb8cd7dd1ec980a639 not found: ID does not exist"
Oct 31 00:53:10.290338 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.290309    4466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"]
Oct 31 00:53:10.299930 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.299907    4466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-a3c14c-openshift-default-5b7bd5fc65-bztx4"]
Oct 31 00:53:10.825605 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:10.825574    4466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" path="/var/lib/kubelet/pods/c09b24d3-dae8-4d07-a741-a0aedc6f6bf0/volumes"
Oct 31 00:53:25.704759 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:25.704721    4466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x28pp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:53:25.705462 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:53:25.705431    4466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x28pp" podUID="07876d43-bffb-4258-a116-fdcdb3e0da8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:53:38.316232 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:53:38.316196    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:54:08.316592 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:54:08.316548    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:54:28.935471 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:28.935436    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86cfd56d94-c7bc6_eba33c69-bc96-416a-80a6-199992fccebd/manager/0.log"
Oct 31 00:54:30.615974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:30.615935    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c745b59c4-mhrbk_3c720be2-27ef-493b-b299-b4c0c4055e36/manager/0.log"
Oct 31 00:54:32.492743 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:32.492717    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7879ff96d6-rthf5_a8d28221-2948-4c1d-999e-d126f1cafe43/maas-api/0.log"
Oct 31 00:54:38.315613 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:54:38.315580    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:54:38.587231 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:38.587153    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546bf7cb95-4gjhl_986a67d7-902f-452a-a839-391beaa99b68/kube-auth-proxy/0.log"
Oct 31 00:54:39.045007 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:39.044976    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-2xfw2_2993c03e-0c6b-4437-b94c-b997e3059d91/istio-proxy/0.log"
Oct 31 00:54:40.188264 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:40.188231    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-875hj_ec7a617c-9fc5-49ec-a02f-d3e7af189f1a/istio-proxy/0.log"
Oct 31 00:54:41.117646 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:41.117615    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-x28pp_07876d43-bffb-4258-a116-fdcdb3e0da8c/istio-proxy/0.log"
Oct 31 00:54:41.400592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:41.400514    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-zgt8d_3941e2c5-941a-41d5-8494-c20c46758469/istio-proxy/0.log"
Oct 31 00:54:43.543356 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:43.543330    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr_c28a2c98-8e52-474d-94c8-5441acdb2954/main/0.log"
Oct 31 00:54:44.102385 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:44.102358    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-274g7_c4d60297-7ba5-4d8a-8c01-0a75a61b0e48/main/0.log"
Oct 31 00:54:44.395330 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:44.395261    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-72tdg_253ddce0-a6b3-41bd-91e4-b27b310d2484/main/0.log"
Oct 31 00:54:44.693062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:44.692985    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv_a3e63cfe-0fd2-4690-b98d-8e106e6bf421/main/0.log"
Oct 31 00:54:44.947080 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:44.946998    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-44s69_ff7fc568-6cf9-48da-960b-860b36f5bc3f/main/0.log"
Oct 31 00:54:45.187496 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:45.187467    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg_d6c3372c-3895-4d31-a94a-b7512c346169/main/0.log"
Oct 31 00:54:45.427280 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:45.427258    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w_beae846e-f30e-48a7-93c0-04fae50b694c/main/0.log"
Oct 31 00:54:56.613421 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:56.613390    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86cfd56d94-c7bc6_eba33c69-bc96-416a-80a6-199992fccebd/manager/0.log"
Oct 31 00:54:57.934608 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:57.934575    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c745b59c4-mhrbk_3c720be2-27ef-493b-b299-b4c0c4055e36/manager/0.log"
Oct 31 00:54:59.277288 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:54:59.277261    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7879ff96d6-rthf5_a8d28221-2948-4c1d-999e-d126f1cafe43/maas-api/0.log"
Oct 31 00:55:04.820129 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:04.820095    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546bf7cb95-4gjhl_986a67d7-902f-452a-a839-391beaa99b68/kube-auth-proxy/0.log"
Oct 31 00:55:05.283374 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:05.283340    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-2xfw2_2993c03e-0c6b-4437-b94c-b997e3059d91/istio-proxy/0.log"
Oct 31 00:55:06.454062 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:06.454028    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-875hj_ec7a617c-9fc5-49ec-a02f-d3e7af189f1a/istio-proxy/0.log"
Oct 31 00:55:07.344904 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:07.344877    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-x28pp_07876d43-bffb-4258-a116-fdcdb3e0da8c/istio-proxy/0.log"
Oct 31 00:55:07.615257 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:07.615187    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-zgt8d_3941e2c5-941a-41d5-8494-c20c46758469/istio-proxy/0.log"
Oct 31 00:55:08.317802 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:55:08.315675    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:55:09.526781 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:09.526752    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-c4dtr_c28a2c98-8e52-474d-94c8-5441acdb2954/main/0.log"
Oct 31 00:55:10.085160 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:10.085133    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-274g7_c4d60297-7ba5-4d8a-8c01-0a75a61b0e48/main/0.log"
Oct 31 00:55:10.363081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:10.363057    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-72tdg_253ddce0-a6b3-41bd-91e4-b27b310d2484/main/0.log"
Oct 31 00:55:10.633469 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:10.633402    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xglqv_a3e63cfe-0fd2-4690-b98d-8e106e6bf421/main/0.log"
Oct 31 00:55:10.861976 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:10.861950    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-44s69_ff7fc568-6cf9-48da-960b-860b36f5bc3f/main/0.log"
Oct 31 00:55:11.105231 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:11.105202    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hnkjg_d6c3372c-3895-4d31-a94a-b7512c346169/main/0.log"
Oct 31 00:55:11.338269 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:11.338241    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-w747w_beae846e-f30e-48a7-93c0-04fae50b694c/main/0.log"
Oct 31 00:55:38.315804 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:55:38.315768    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:55:41.884909 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:41.884880    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c745b59c4-mhrbk_3c720be2-27ef-493b-b299-b4c0c4055e36/manager/0.log"
Oct 31 00:55:42.465314 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:55:42.465285    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c745b59c4-mhrbk_3c720be2-27ef-493b-b299-b4c0c4055e36/manager/0.log"
Oct 31 00:56:08.315451 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:56:08.315410    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:56:24.825027 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:24.824993    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c745b59c4-mhrbk_3c720be2-27ef-493b-b299-b4c0c4055e36/manager/0.log"
Oct 31 00:56:25.406940 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:25.406904    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c745b59c4-mhrbk_3c720be2-27ef-493b-b299-b4c0c4055e36/manager/0.log"
Oct 31 00:56:26.919588 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919560    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5nxv/must-gather-bzd47"]
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919911    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ab1f6566-933e-4e88-aa17-9cbfe8e722b3" containerName="maas-api"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919923    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1f6566-933e-4e88-aa17-9cbfe8e722b3" containerName="maas-api"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919939    4466 cpu_manager.go:414] "containerMap: removing container" podUID="ebba3459-19c3-4bf2-9e18-2db0b5a29327" containerName="payload-pre-processing"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919943    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebba3459-19c3-4bf2-9e18-2db0b5a29327" containerName="payload-pre-processing"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919952    4466 cpu_manager.go:414] "containerMap: removing container" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919957    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919964    4466 cpu_manager.go:414] "containerMap: removing container" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy"
Oct 31 00:56:26.919974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.919968    4466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy"
Oct 31 00:56:26.920221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.920016    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1f6566-933e-4e88-aa17-9cbfe8e722b3" containerName="maas-api"
Oct 31 00:56:26.920221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.920026    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebba3459-19c3-4bf2-9e18-2db0b5a29327" containerName="payload-pre-processing"
Oct 31 00:56:26.920221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.920032    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy"
Oct 31 00:56:26.920221 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.920038    4466 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09b24d3-dae8-4d07-a741-a0aedc6f6bf0" containerName="istio-proxy"
Oct 31 00:56:26.923047 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:26.923027    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.012592 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.012557    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1a107ff-356d-4137-a96c-aac36951c1fb-must-gather-output\") pod \"must-gather-bzd47\" (UID: \"e1a107ff-356d-4137-a96c-aac36951c1fb\") " pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.012748 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.012605    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nkx\" (UniqueName: \"kubernetes.io/projected/e1a107ff-356d-4137-a96c-aac36951c1fb-kube-api-access-86nkx\") pod \"must-gather-bzd47\" (UID: \"e1a107ff-356d-4137-a96c-aac36951c1fb\") " pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.030447 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.030422    4466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5nxv\"/\"default-dockercfg-k229d\""
Oct 31 00:56:27.030447 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.030422    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5nxv\"/\"openshift-service-ca.crt\""
Oct 31 00:56:27.030619 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.030486    4466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5nxv\"/\"kube-root-ca.crt\""
Oct 31 00:56:27.032069 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.032051    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/must-gather-bzd47"]
Oct 31 00:56:27.113457 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.113431    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1a107ff-356d-4137-a96c-aac36951c1fb-must-gather-output\") pod \"must-gather-bzd47\" (UID: \"e1a107ff-356d-4137-a96c-aac36951c1fb\") " pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.113594 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.113462    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nkx\" (UniqueName: \"kubernetes.io/projected/e1a107ff-356d-4137-a96c-aac36951c1fb-kube-api-access-86nkx\") pod \"must-gather-bzd47\" (UID: \"e1a107ff-356d-4137-a96c-aac36951c1fb\") " pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.113822 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.113804    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1a107ff-356d-4137-a96c-aac36951c1fb-must-gather-output\") pod \"must-gather-bzd47\" (UID: \"e1a107ff-356d-4137-a96c-aac36951c1fb\") " pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.176828 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.176768    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nkx\" (UniqueName: \"kubernetes.io/projected/e1a107ff-356d-4137-a96c-aac36951c1fb-kube-api-access-86nkx\") pod \"must-gather-bzd47\" (UID: \"e1a107ff-356d-4137-a96c-aac36951c1fb\") " pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.235951 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.235914    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/must-gather-bzd47"
Oct 31 00:56:27.602049 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:56:27.602013    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1a107ff_356d_4137_a96c_aac36951c1fb.slice/crio-875e85c956652a641abd33e87cc39487550bf6b61c9e9ce0bdb6c5179bf1cd7d WatchSource:0}: Error finding container 875e85c956652a641abd33e87cc39487550bf6b61c9e9ce0bdb6c5179bf1cd7d: Status 404 returned error can't find the container with id 875e85c956652a641abd33e87cc39487550bf6b61c9e9ce0bdb6c5179bf1cd7d
Oct 31 00:56:27.603899 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.603883    4466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:56:27.604724 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:27.604704    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/must-gather-bzd47"]
Oct 31 00:56:28.127174 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:28.127145    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/must-gather-bzd47" event={"ID":"e1a107ff-356d-4137-a96c-aac36951c1fb","Type":"ContainerStarted","Data":"875e85c956652a641abd33e87cc39487550bf6b61c9e9ce0bdb6c5179bf1cd7d"}
Oct 31 00:56:29.135796 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:29.135755    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/must-gather-bzd47" event={"ID":"e1a107ff-356d-4137-a96c-aac36951c1fb","Type":"ContainerStarted","Data":"b13e6d830535db4c8a56573a7de7db37440ae6941e41d7e2b5ecd0588020441c"}
Oct 31 00:56:30.142322 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:30.142281    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/must-gather-bzd47" event={"ID":"e1a107ff-356d-4137-a96c-aac36951c1fb","Type":"ContainerStarted","Data":"5e543670b76264331132d1dff3d1e161235049b88db4a1f1ce177078ea3e1214"}
Oct 31 00:56:30.264613 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:30.264550    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q5nxv/must-gather-bzd47" podStartSLOduration=2.910513157 podStartE2EDuration="4.264536762s" totalImagesPullingTime="1.354023605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:30.262264028 +0000 UTC m=+4495.552812347" watchObservedRunningTime="2026-10-31 00:56:30.264536762 +0000 UTC m=+4495.555085078"
Oct 31 00:56:37.605076 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:37.603209    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gl7bk_45dfef13-d5e6-4bff-98d2-47032ef6f624/global-pull-secret-syncer/0.log"
Oct 31 00:56:38.381655 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:56:38.380685    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:56:40.159552 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:40.159164    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d7f6p_e895adf2-43c1-46f2-b2de-a3bddfd08f0c/konnectivity-agent/0.log"
Oct 31 00:56:40.948812 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:40.948782    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-123.us-west-2.compute.internal_e32b577676385ef454846283b75daa6e/haproxy/0.log"
Oct 31 00:56:45.681725 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.681675    4466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"]
Oct 31 00:56:45.686169 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.686149    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.716122 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.716094    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-sys\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.716298 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.716272    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-podres\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.716398 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.716334    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-lib-modules\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.716470 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.716414    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-proc\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.716531 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.716503    4466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlsn\" (UniqueName: \"kubernetes.io/projected/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-kube-api-access-5nlsn\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.775094 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.775059    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"]
Oct 31 00:56:45.817921 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.817894    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-podres\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.817936    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-lib-modules\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.817953    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-proc\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.817973    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlsn\" (UniqueName: \"kubernetes.io/projected/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-kube-api-access-5nlsn\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.818001    4466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-sys\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.818044    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-lib-modules\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.818056    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-proc\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.818043    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-podres\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.818105 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.818089    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-sys\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:45.881688 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:45.881658    4466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlsn\" (UniqueName: \"kubernetes.io/projected/f4b3d3c8-3c83-4baa-a998-b669c1ee19b6-kube-api-access-5nlsn\") pod \"perf-node-gather-daemonset-q7mlj\" (UID: \"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:46.008304 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:46.008212    4466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:46.385114 ip-10-0-131-123 kubenswrapper[4466]: W1031 00:56:46.385080    4466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4b3d3c8_3c83_4baa_a998_b669c1ee19b6.slice/crio-1321ed6e0a66399f436eed8091744db7824949fbe5274eed174a6e5e7337a062 WatchSource:0}: Error finding container 1321ed6e0a66399f436eed8091744db7824949fbe5274eed174a6e5e7337a062: Status 404 returned error can't find the container with id 1321ed6e0a66399f436eed8091744db7824949fbe5274eed174a6e5e7337a062
Oct 31 00:56:46.387350 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:46.387327    4466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"]
Oct 31 00:56:47.253128 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:47.253085    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj" event={"ID":"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6","Type":"ContainerStarted","Data":"e77124e9d417ea524cc3e468c397139c033d961147d283a0457a2214f45fb7e4"}
Oct 31 00:56:47.253519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:47.253188    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:47.253519 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:47.253214    4466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj" event={"ID":"f4b3d3c8-3c83-4baa-a998-b669c1ee19b6","Type":"ContainerStarted","Data":"1321ed6e0a66399f436eed8091744db7824949fbe5274eed174a6e5e7337a062"}
Oct 31 00:56:47.374134 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:47.374086    4466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj" podStartSLOduration=2.374074444 podStartE2EDuration="2.374074444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:47.373446467 +0000 UTC m=+4512.663994784" watchObservedRunningTime="2026-10-31 00:56:47.374074444 +0000 UTC m=+4512.664622762"
Oct 31 00:56:53.266974 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:53.266941    4466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-q7mlj"
Oct 31 00:56:55.299766 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:55.299725    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d948c677c-pt8sl_2dca4577-7118-4c8c-a6f9-fb72f7b386ad/metrics-server/0.log"
Oct 31 00:56:55.702537 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:55.702507    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-kff85_eda1ddc3-ab20-471e-b273-9fa66556e35f/monitoring-plugin/0.log"
Oct 31 00:56:57.379229 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:57.379174    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slvxx_d48dc9d1-11d0-4633-9cf7-50f4229fdc2e/node-exporter/0.log"
Oct 31 00:56:57.792732 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:57.792655    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slvxx_d48dc9d1-11d0-4633-9cf7-50f4229fdc2e/kube-rbac-proxy/0.log"
Oct 31 00:56:58.185928 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:56:58.185900    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slvxx_d48dc9d1-11d0-4633-9cf7-50f4229fdc2e/init-textfile/0.log"
Oct 31 00:57:04.580176 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:57:04.580145    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-t8wch_84833d2f-e775-46d1-be8c-a56a1c9c0b0a/prometheus-operator-admission-webhook/0.log"
Oct 31 00:57:08.315178 ip-10-0-131-123 kubenswrapper[4466]: E1031 00:57:08.315139    4466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cm5bp" containerName="extractor"
Oct 31 00:57:09.687847 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:57:09.687817    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546bf7cb95-4gjhl_986a67d7-902f-452a-a839-391beaa99b68/kube-auth-proxy/0.log"
Oct 31 00:57:10.099081 ip-10-0-131-123 kubenswrapper[4466]: I1031 00:57:10.099033    4466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-2xfw2_2993c03e-0c6b-4437-b94c-b997e3059d91/istio-proxy/0.log"