Oct 30 18:37:56.515843 ip-10-0-134-133 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:37:56.898543 ip-10-0-134-133 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:56.898543 ip-10-0-134-133 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:37:56.898543 ip-10-0-134-133 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:56.898543 ip-10-0-134-133 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:56.901824 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901691    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.901824 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901824    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901828    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901832    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901836    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901839    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901842    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901844    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901847    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901850    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901853    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901856    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901859    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901861    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901864    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901867    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901869    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901872    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901875    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901883    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901886    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.901895 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901889    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901892    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901895    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901897    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901900    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901903    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901906    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901909    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901912    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901914    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901917    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901919    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901922    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901924    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901927    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901929    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901932    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901935    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901937    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901940    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.902351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901943    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901945    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901948    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901950    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901953    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901955    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901958    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901960    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901963    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901965    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901968    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901971    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901974    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901977    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901980    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901983    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901985    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901988    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901990    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901993    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.902830 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901995    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.901998    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902000    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902003    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902005    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902008    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902011    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902013    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902017    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902021    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902025    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902028    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902031    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902034    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902036    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902039    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902043    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902047    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.903293 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902050    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902052    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902055    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902057    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902060    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902063    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902066    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902069    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902073    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902075    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902078    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902081    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902083    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902086    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902088    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902091    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902094    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902096    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902099    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902101    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.903743 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902104    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902107    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902109    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902111    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902114    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902117    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902120    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902908    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902916    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902919    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902923    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902926    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902929    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902932    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902935    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902937    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902940    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902943    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902945    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902949    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.904864 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902952    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902955    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902958    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902960    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902963    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902965    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902968    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902971    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902974    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902977    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902979    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902982    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902985    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902988    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902990    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902993    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902995    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.902998    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903001    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903003    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.905351 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903006    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903008    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903011    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903014    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903017    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903019    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903022    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903025    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903027    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903031    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903036    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903039    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903042    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903045    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903047    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903050    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903052    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903055    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903058    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.905843 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903061    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903063    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903066    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903069    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903071    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903074    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903076    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903079    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903081    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903085    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903087    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903090    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903093    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903096    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903098    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903101    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903104    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903107    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903109    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903112    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.906294 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903114    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903117    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903120    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903125    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903129    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903132    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903135    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903137    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903140    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903143    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903145    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903148    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903151    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903153    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903156    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903159    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903161    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903164    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903166    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.906781 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903169    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903172    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903174    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903177    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903179    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903182    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903184    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903187    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903190    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903193    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903195    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903198    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903200    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903203    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903205    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903694    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903701    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903704    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903706    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903712    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.907249 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903715    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903720    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903724    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903728    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903731    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903734    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903737    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903740    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903743    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903746    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903749    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903752    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903754    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903757    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903760    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903762    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903765    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903768    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903771    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.907737 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903773    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903776    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903778    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903781    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903783    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903786    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903789    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903791    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903794    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903798    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903801    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903803    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903807    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903811    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903814    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903817    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903819    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903822    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903825    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.908223 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903827    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903830    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903832    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903835    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903837    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903840    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903842    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903845    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903847    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903850    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903853    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903855    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903858    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903860    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903863    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903865    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903868    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903870    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903873    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.908719 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903875    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903878    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903881    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903884    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903887    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903889    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903894    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903896    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903900    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903902    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903906    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903910    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903913    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903916    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903919    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903921    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903924    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903926    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903929    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903932    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.909156 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903934    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903937    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903939    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903942    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903944    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903947    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903950    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903952    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903955    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903957    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903959    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903962    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903964    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903967    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903969    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903973    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903976    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903978    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903983    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903985    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.909691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903989    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903992    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903994    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.903997    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904745    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904754    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904761    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904765    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904770    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904773    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904781    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904785    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904789    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904792    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904795    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904799    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904802    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904805    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904808    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904811    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904814    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904817    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904822    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:37:56.910162 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904825    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904828    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904831    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904835    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904840    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904843    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904847    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904851    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904854    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904857    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904861    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904865    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904868    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904873    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904876    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904879    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904882    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904885    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904888    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904892    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904895    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904898    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904902    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904905    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:37:56.910695 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904909    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904912    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904915    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904918    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904922    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904925    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904928    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904931    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904934    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904937    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904940    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904944    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904948    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904951    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904954    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904957    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904960    2580 flags.go:64] FLAG: --help="false"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904963    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904967    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904970    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904973    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904977    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904980    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:37:56.911316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904983    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904986    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904989    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904992    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904995    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.904998    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905001    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905003    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905006    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905010    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905013    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905015    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905018    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905021    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905024    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905029    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905032    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905035    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905038    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905042    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905045    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905048    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905051    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905056    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905059    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:37:56.912080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905063    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905066    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905069    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905073    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905076    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905079    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905082    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905085    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905092    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905095    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905098    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905101    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905104    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905107    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905110    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905113    2580 flags.go:64] FLAG: --port="10250"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905116    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905119    2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d20d98005a218369"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905122    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905125    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905129    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905131    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905136    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905138    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905141    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905144    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:37:56.913040 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905148    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905151    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905154    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905157    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905160    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905163    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905166    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905169    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905172    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905175    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905185    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905189    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905193    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905196    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905199    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905202    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905205    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905208    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905211    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905214    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905219    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905222    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905225    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905230    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905233    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:37:56.913743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905235    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905238    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905241    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905244    2580 flags.go:64] FLAG: --v="2"
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905248    2580 flags.go:64] FLAG: --version="false"
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905252    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905256    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905260    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905399    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905404    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905407    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905409    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905412    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905416    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905420    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905424    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905427    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905429    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905432    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905434    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905438    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.914316 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905440    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905443    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905445    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905448    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905451    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905453    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905456    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905459    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905461    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905464    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905466    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905469    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905471    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905474    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905476    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905480    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905483    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905485    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905488    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.914832 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905491    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905497    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905499    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905503    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905505    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905508    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905511    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905514    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905516    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905519    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905521    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905524    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905527    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905530    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905533    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905535    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905538    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905541    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905543    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905546    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.915280 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905548    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905551    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905554    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905556    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905559    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905561    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905564    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905566    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905570    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905573    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905575    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905593    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905597    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905603    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905607    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905610    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905612    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905615    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905618    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905621    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.915755 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905623    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905626    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905629    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905631    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905634    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905637    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905640    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905643    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905645    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905648    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905651    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905653    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905656    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905658    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905661    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905663    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905666    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905668    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905671    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905673    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.916220 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905677    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905680    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905682    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905685    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905689    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905694    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905697    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905700    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905703    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905706    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905708    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905711    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905713    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905716    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.905743    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905783    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.916773 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905788    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905791    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905794    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905798    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905801    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905803    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905806    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905808    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905811    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905814    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905816    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905819    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905821    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905824    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905827    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905829    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905832    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905837    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905840    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905842    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.917190 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905845    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905849    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905852    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905855    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905858    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905860    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905863    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905866    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905868    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905871    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905874    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905876    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905879    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905882    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905884    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905887    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905890    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905893    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905896    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905898    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.917692 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905901    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905903    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905906    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905908    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905910    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905913    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905915    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905918    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905921    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905924    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905927    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905930    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905932    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905936    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905938    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905941    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905943    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905946    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905949    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.918155 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905951    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905954    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905957    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905959    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905962    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905965    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905967    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905970    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905972    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905975    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905978    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905980    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905983    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905985    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905988    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905990    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905993    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905995    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.905998    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.918600 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906000    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906004    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906007    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906011    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906015    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906017    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906020    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906024    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906027    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906029    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906032    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906034    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906037    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906040    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906042    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906045    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906049    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906053    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906056    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.919097 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906058    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906061    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906063    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906066    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906069    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906072    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906074    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.906077    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.913365    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.913379    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.913393    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.913398    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913454    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913459    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913463    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913466    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913470    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913472    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913475    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.919539 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913478    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913481    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913484    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913487    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913490    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913492    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913495    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913498    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913500    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913503    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913506    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913508    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913511    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913514    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913517    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913520    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913523    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913526    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913529    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.920006 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913531    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913534    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913536    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913539    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913543    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913547    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913550    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913553    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913556    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913559    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913561    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913564    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913567    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913570    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913572    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913575    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913595    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913599    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913602    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.920459 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913604    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913608    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913611    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913614    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913617    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913620    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913622    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913625    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913627    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913630    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913632    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913637    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913640    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913644    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913647    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913649    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913652    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913654    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913657    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913660    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.921049 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913662    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913665    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913667    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913670    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913672    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913675    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913677    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913680    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913682    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913685    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913687    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913690    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913692    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913695    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913699    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913701    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913704    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913707    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913709    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913712    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.921507 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913715    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913717    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913720    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913722    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913725    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913727    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913730    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913732    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913735    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913737    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913740    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913742    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913745    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913747    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913750    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913753    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913756    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913759    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913762    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913764    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.922067 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913767    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.913796    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913933    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913938    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913941    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913944    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913947    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913951    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913953    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913956    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913958    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913961    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913965    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913968    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913971    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913974    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.922557 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913977    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913979    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913982    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913984    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913987    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913989    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913992    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913994    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.913997    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914000    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914002    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914005    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914008    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914010    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914013    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914015    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914017    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914020    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914022    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914025    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.922953 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914028    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914033    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914035    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914038    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914041    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914044    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914046    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914050    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914054    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914057    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914060    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914063    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914066    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914069    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914071    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914074    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914076    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914079    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914081    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.923433 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914084    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914086    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914089    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914092    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914094    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914096    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914100    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914103    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914106    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914109    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914111    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914114    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914116    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914119    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914123    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914126    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914128    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914131    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914134    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.923887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914137    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914139    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914142    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914145    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914147    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914150    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914152    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914155    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914158    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914160    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914163    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914165    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914168    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914170    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914173    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914175    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914178    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914181    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914183    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914187    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.924340 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914190    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914192    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914195    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914198    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914200    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914203    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914205    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914209    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914211    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914214    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914217    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914219    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914222    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914225    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.914248    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:56.924855 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914278    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914282    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914287    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914290    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914293    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914296    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914298    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914301    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914303    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914306    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914309    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914311    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914314    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914317    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914320    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914324    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914327    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914330    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914332    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:56.925206 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914335    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914338    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914341    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914343    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914346    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914348    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914351    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914355    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914357    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914360    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914362    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914365    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914368    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914372    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914376    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914378    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914381    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914384    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914386    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914390    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:56.925691 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914392    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914395    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914397    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914400    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914402    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914405    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914407    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914410    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914412    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914415    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914418    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914421    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914423    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914426    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914429    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914431    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914434    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914436    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914439    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:56.926229 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914442    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914444    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914447    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914449    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914452    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914455    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914458    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914460    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914463    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914465    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914468    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914470    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914473    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914475    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914478    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914481    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914483    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914486    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914488    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914490    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:56.926842 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914493    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914495    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914498    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914501    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914503    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914506    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914508    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914511    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914513    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914515    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914518    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914520    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914523    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914525    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914528    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914530    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914533    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914535    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914538    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914541    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:56.927416 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914543    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914546    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914549    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914551    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914554    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914557    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914560    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:56.914562    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.914632    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.916484    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.918831    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.918930    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.920267    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.920814    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:37:56.927917 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.921863    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:37:56.933287 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.933271    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:37:56.936119 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.936103    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 18:37:56.936909 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.936888    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:37:56.941126 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.941105    2580 fs.go:135] Filesystem UUIDs: map[541265af-cc59-44b0-a9ad-2826cd2f94a9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b7c8123e-01b2-4a3e-9a6b-85d85a09d367:/dev/nvme0n1p3]
Oct 30 18:37:56.941207 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.941126    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:37:56.947656 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.947525    2580 manager.go:217] Machine: {Timestamp:2026-10-30 18:37:56.9462864 +0000 UTC m=+0.317861494 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182730 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec225ff72f206814640b025affea46b2 SystemUUID:ec225ff7-2f20-6814-640b-025affea46b2 BootID:de56a7de-4a2c-4141-9ade-9254a26a2001 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d9:ca:9a:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d9:ca:9a:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:92:6c:ff:30:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:37:56.947656 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.947647    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:37:56.947822 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.947742    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:37:56.948641 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.948622    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:37:56.948764 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.948641    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-133.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:37:56.948811 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.948766    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:37:56.948811 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.948774    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:37:56.948860 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.948813    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:37:56.948860 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.948830    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:37:56.949613 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.949602    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:37:56.953837 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.953825    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:37:56.954000 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.953990    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:37:56.954707 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.954697    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:37:56.954738 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.954710    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:37:56.954738 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.954720    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:37:56.954738 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.954730    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:37:56.954738 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.954737    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:37:56.955048 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.955008    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:37:56.959268 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.959249    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:37:56.963519 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.963506    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:37:56.964332 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964317    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:37:56.964367 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964355    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:37:56.964516 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964504    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:37:56.964548 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964524    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:37:56.964548 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964530    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:37:56.964548 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964536    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:37:56.964548 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964541    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:37:56.964548 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.964547    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:37:56.965037 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.965028    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:37:56.965065 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.965038    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:37:56.965065 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.965045    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:37:56.965065 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.965051    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:37:56.965065 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.965059    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:37:56.965217 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.965068    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:37:56.966193 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.966182    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:37:56.966230 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.966196    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:37:56.969769 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.969754    2580 server.go:1263] "Started kubelet"
Oct 30 18:37:56.970088 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.969827    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:37:56.970190 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.969870    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:37:56.970245 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.970188    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:37:56.970480 ip-10-0-134-133 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:37:56.971157 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.971139    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:37:56.972782 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.972767    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:37:56.975373 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.975354    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:37:56.975373 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.975366    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:56.975489 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:56.975400    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:37:56.975489 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:56.975451    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:56.975928 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.975911    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:37:56.976045 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.975914    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:37:56.976120 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976048    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:37:56.976120 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976114    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:37:56.976204 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976122    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:37:56.976234 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976215    2580 factory.go:55] Registering systemd factory
Oct 30 18:37:56.976234 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976228    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:37:56.976489 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976465    2580 factory.go:153] Registering CRI-O factory
Oct 30 18:37:56.976489 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976484    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:37:56.976627 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976530    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:37:56.976627 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976548    2580 factory.go:103] Registering Raw factory
Oct 30 18:37:56.976627 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.976563    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:37:56.977684 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.977665    2580 manager.go:319] Starting recovery of all containers
Oct 30 18:37:56.990935 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.990829    2580 manager.go:324] Recovery completed
Oct 30 18:37:56.994669 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.994657    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:56.996364 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996342    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:56.996415 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996376    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:56.996415 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996386    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:56.996923 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996908    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:37:56.996923 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996921    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:37:56.997023 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996937    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:37:56.997023 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.996914    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:56.997655 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.997640    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:56.997692 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.997669    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:56.997692 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.997679    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:56.999142 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.999130    2580 policy_none.go:50] "Start"
Oct 30 18:37:56.999187 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.999150    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:37:56.999187 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:56.999161    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:37:57.001156 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.001146    2580 policy_none.go:44] "Start"
Oct 30 18:37:57.054141 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.054112    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:37:57.056486 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.056460    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:37:57.056571 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.056506    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:37:57.056571 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.056518    2580 server.go:87] "Starting device plugin registration server"
Oct 30 18:37:57.056765 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.056752    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:37:57.056816 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.056767    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:37:57.057074 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.056905    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:37:57.057074 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.057003    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:37:57.057074 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.057014    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:37:57.057543 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.057521    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:37:57.057643 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.057611    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:57.119171 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.119144    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-133.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:57.121049 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.119141    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-133.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:37:57.121049 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.119152    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-133.us-west-2.compute.internal.18e36296ec1ffe1c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-133.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-133.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:56.969725468 +0000 UTC m=+0.341300540,LastTimestamp:2026-10-30 18:37:56.969725468 +0000 UTC m=+0.341300540,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-133.us-west-2.compute.internal,}"
Oct 30 18:37:57.126496 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.126472    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:37:57.127896 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.127881    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:37:57.127982 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.127903    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:37:57.127982 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.127924    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:37:57.127982 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.127960    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:37:57.157895 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.157851    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:57.158542 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.158526    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:57.158619 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.158563    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:57.158619 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.158576    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:57.158696 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.158631    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.211470 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.211452    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:57.211575 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.211494    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bf7w8"
Oct 30 18:37:57.224621 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.224600    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bf7w8"
Oct 30 18:37:57.228558 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.228539    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"]
Oct 30 18:37:57.228625 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.228612    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:57.229564 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.229548    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:57.229652 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.229603    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:57.229652 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.229614    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:57.231721 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.231709    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:57.231900 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.231885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.231939 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.231917    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:57.232410 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.232396    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:57.232457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.232405    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:57.232457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.232425    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:57.232457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.232436    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:57.232457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.232443    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:57.232457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.232455    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:57.234454 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.234440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.234499 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.234467    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:57.235114 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.235101    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:57.235185 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.235130    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:57.235185 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.235144    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:57.260100 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.260084    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.262577 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.262560    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.272122 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.272103    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:37:57.272201 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.272182    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:37:57.272316 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.272295    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-133.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:37:57.272316 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.272229    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-133.us-west-2.compute.internal.18e36296edb67799  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-133.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-133.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-133.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:56.996364185 +0000 UTC m=+0.367939259,LastTimestamp:2026-10-30 18:37:56.996364185 +0000 UTC m=+0.367939259,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-133.us-west-2.compute.internal,}"
Oct 30 18:37:57.280948 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.280928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6d9ef131d394299abae730c3d467d74-config\") pod \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"c6d9ef131d394299abae730c3d467d74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.281015 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.280956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.281015 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.280984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.381335 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.381314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6d9ef131d394299abae730c3d467d74-config\") pod \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"c6d9ef131d394299abae730c3d467d74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.381463 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.381340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.381463 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.381359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.381463 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.381387    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.381463 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.381409    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6d9ef131d394299abae730c3d467d74-config\") pod \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"c6d9ef131d394299abae730c3d467d74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.381463 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.381420    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.426156 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.426107    2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-133.us-west-2.compute.internal\" not found" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.426226 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.426211    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:57.474977 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.474959    2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-133.us-west-2.compute.internal\" not found" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.563454 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.563442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.565496 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.565481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.626941 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.626921    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:57.627751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.627737    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:57.627824 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.627764    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:57.627824 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.627776    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:57.627824 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.627801    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.682016 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.681976    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:57.682016 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.681994    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-133.us-west-2.compute.internal\": node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:57.785967 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.785938    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:57.886188 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.886157    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:57.917892 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.917854    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:37:57.918438 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:57.917999    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:37:57.918438 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.918010    2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a97600e569e5c4a9eafb16c9ff6ff0f2-36f3c3322d66c93d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-133.us-west-2.compute.internal.18e36296edb6dcce\": read tcp 10.0.134.133:60078->3.131.2.21:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-133.us-west-2.compute.internal.18e36296edb6dcce  default   5078 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-133.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-133.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-133.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:56 +0000 UTC,LastTimestamp:2026-10-30 18:37:57.158596762 +0000 UTC m=+0.530171835,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-133.us-west-2.compute.internal,}"
Oct 30 18:37:57.986780 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:57.986753    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.087444 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.087421    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.173013 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.172994    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:58.173963 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.173948    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:58.174054 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.173975    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:58.174054 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.173992    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:58.187913 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.187892    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.229392 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.229362    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:57 +0000 UTC" deadline="2028-07-09 07:58:37.912139794 +0000 UTC"
Oct 30 18:37:58.229392 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.229386    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14821h20m39.682756863s"
Oct 30 18:37:58.288854 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.288804    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.389111 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.389086    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.489286 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.489260    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.589717 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.589687    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.689835 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:58.689809    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:37:58.690408 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:58.690386    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47490914e2d482394d4603a5e95f123.slice/crio-f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3 WatchSource:0}: Error finding container f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3: Status 404 returned error can't find the container with id f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3
Oct 30 18:37:58.690822 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:58.690809    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d9ef131d394299abae730c3d467d74.slice/crio-3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55 WatchSource:0}: Error finding container 3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55: Status 404 returned error can't find the container with id 3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55
Oct 30 18:37:58.695484 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.695461    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:37:58.717105 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.717066    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:58.717210 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.717104    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:58.717210 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.717104    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:58.776129 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.776108    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:58.838271 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.838211    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:58.839373 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.839358    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:37:58.902524 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.902504    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:58.955830 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.955797    2580 apiserver.go:52] "Watching apiserver"
Oct 30 18:37:58.977671 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:58.977649    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:59.069604 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.069561    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:37:59.069955 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.069931    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-rvg7d","kube-system/global-pull-secret-syncer-77qht","openshift-image-registry/node-ca-7vv9s","openshift-multus/multus-n4t4x","openshift-multus/network-metrics-daemon-qd6bc","openshift-cluster-node-tuning-operator/tuned-4md92","openshift-dns/node-resolver-s4ps2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal","kube-system/konnectivity-agent-fdf4b","kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-shstn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc","openshift-multus/multus-additional-cni-plugins-4ljqw","openshift-network-diagnostics/network-check-target-mdb7t"]
Oct 30 18:37:59.073989 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.073967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.076234 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.076211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.078409 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.078383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.080697 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.080677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.082892 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.082876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.082984 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.082947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:37:59.085741 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.085726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.088095 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.088075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.089386 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e510aafb-e0d8-4898-a6ac-6aaaac776987-host\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.089386 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7brm\" (UniqueName: \"kubernetes.io/projected/f1fea5aa-6e82-4768-8f35-fb947216fb9f-kube-api-access-j7brm\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.089492 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089492 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-cnibin\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089610 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089610 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e510aafb-e0d8-4898-a6ac-6aaaac776987-serviceca\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.089743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1fea5aa-6e82-4768-8f35-fb947216fb9f-hosts-file\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.089743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fea5aa-6e82-4768-8f35-fb947216fb9f-tmp-dir\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.089819 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-system-cni-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089819 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089819 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4sb\" (UniqueName: \"kubernetes.io/projected/cd48639e-7f2c-4383-9716-29c1b4e68994-kube-api-access-pm4sb\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089909 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.089939 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.089909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfml5\" (UniqueName: \"kubernetes.io/projected/e510aafb-e0d8-4898-a6ac-6aaaac776987-kube-api-access-hfml5\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.090355 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.090339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.094326 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.094300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.094418 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.094373    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:37:59.096887 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.096868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:37:59.096985 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.096922    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:37:59.099234 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.099217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.101517 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.101496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.131761 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.131733    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.131883 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.131758    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.131883 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.131794    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-86nv8\""
Oct 30 18:37:59.132341 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.132306    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"c6d9ef131d394299abae730c3d467d74","Type":"ContainerStarted","Data":"3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55"}
Oct 30 18:37:59.133188 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133175    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:37:59.133327 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133316    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4gd2\""
Oct 30 18:37:59.133568 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133549    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.133717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133570    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"d47490914e2d482394d4603a5e95f123","Type":"ContainerStarted","Data":"f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3"}
Oct 30 18:37:59.133717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133622    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.133717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133684    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:37:59.133717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.133692    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:37:59.135127 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.135113    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.135240 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.135229    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.135305 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.135294    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27m54\""
Oct 30 18:37:59.135483 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.135470    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:37:59.137771 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.137759    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:37:59.137852 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.137840    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5c5lj\""
Oct 30 18:37:59.142769 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.142755    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.143912 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.143895    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w46g4\""
Oct 30 18:37:59.178194 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.178178    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:37:59.183962 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.183943    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:59.184011 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.183974    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-thw55"
Oct 30 18:37:59.190552 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-var-lib-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.190625 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.190625 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xtl\" (UniqueName: \"kubernetes.io/projected/fca86dc7-9dba-44fa-af4b-209629079b49-kube-api-access-26xtl\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.190625 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-modprobe-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.190757 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-run\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.190757 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-host\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.190757 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-socket-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.190757 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-sys-fs\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.190757 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.190935 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-etc-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.190935 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-netns\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.190935 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-multus\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.190935 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-conf\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.191073 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt5k\" (UniqueName: \"kubernetes.io/projected/48d98419-9d48-4a09-96cd-940bee92d0b3-kube-api-access-2tt5k\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.191073 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.190966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:37:59.191073 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fea5aa-6e82-4768-8f35-fb947216fb9f-tmp-dir\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.191073 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-system-cni-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfml5\" (UniqueName: \"kubernetes.io/projected/e510aafb-e0d8-4898-a6ac-6aaaac776987-kube-api-access-hfml5\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191159    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-system-cni-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-system-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-systemd\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.191241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-tuned\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191246    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191306    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fea5aa-6e82-4768-8f35-fb947216fb9f-tmp-dir\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-tmp\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e510aafb-e0d8-4898-a6ac-6aaaac776987-serviceca\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-node-log\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-bin\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-conf-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-kubernetes\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtft\" (UniqueName: \"kubernetes.io/projected/0808db1b-d15b-47c9-aa62-8a1a6309398b-kube-api-access-xhtft\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.191582 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-log-socket\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191741    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e510aafb-e0d8-4898-a6ac-6aaaac776987-serviceca\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.191975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-k8s-cni-cncf-io\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-bin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-lib-modules\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7brm\" (UniqueName: \"kubernetes.io/projected/f1fea5aa-6e82-4768-8f35-fb947216fb9f-kube-api-access-j7brm\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-script-lib\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-sys\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-kubelet-config\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-multus-certs\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-agent-certs\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-ovn\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-socket-dir-parent\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-hostroot\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.192295 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192297    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crfc\" (UniqueName: \"kubernetes.io/projected/cfd92d06-4598-48c4-822b-d0dff1df2936-kube-api-access-6crfc\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-systemd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1fea5aa-6e82-4768-8f35-fb947216fb9f-hosts-file\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-slash\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192765    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1fea5aa-6e82-4768-8f35-fb947216fb9f-hosts-file\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.192894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-cnibin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.192961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-cni-binary-copy\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-config\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-var-lib-kubelet\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-registration-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e510aafb-e0d8-4898-a6ac-6aaaac776987-host\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8924fe8-5503-4a1d-a361-18221c829534-host-slash\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.193306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193284    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e510aafb-e0d8-4898-a6ac-6aaaac776987-host\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-kubelet\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-systemd-units\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-netd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-env-overrides\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovn-node-metrics-cert\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-dbus\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-cnibin\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8ch\" (UniqueName: \"kubernetes.io/projected/b8924fe8-5503-4a1d-a361-18221c829534-kube-api-access-cb8ch\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.193735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-netns\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193773    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-cnibin\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4sb\" (UniqueName: \"kubernetes.io/projected/cd48639e-7f2c-4383-9716-29c1b4e68994-kube-api-access-pm4sb\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8924fe8-5503-4a1d-a361-18221c829534-iptables-alerter-script\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.193986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-kubelet\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-etc-kubernetes\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysconfig\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-konnectivity-ca\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.194183 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-daemon-config\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.194560 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-device-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.194560 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.194560 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.194315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbn8\" (UniqueName: \"kubernetes.io/projected/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kube-api-access-crbn8\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.195501 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.195471    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.195652 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.195605    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.239816 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.239791    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:37:59.239816 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.239805    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:37:59.239981 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.239864    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6x86\""
Oct 30 18:37:59.239981 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.239880    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rrn4s\""
Oct 30 18:37:59.239981 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.239867    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:37:59.240137 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240041    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vpq5z\""
Oct 30 18:37:59.240137 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240060    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.240137 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240089    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6fhn6\""
Oct 30 18:37:59.240137 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240123    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:37:59.240287 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240276    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:37:59.240338 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240307    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.240338 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240318    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:37:59.240338 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240335    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:37:59.240491 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240468    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.240534 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240521    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.240664 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240627    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.240664 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240643    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:37:59.240792 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240685    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:59.240792 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.240781    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:37:59.255966 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.255939    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:37:59.259296 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.259270    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfml5\" (UniqueName: \"kubernetes.io/projected/e510aafb-e0d8-4898-a6ac-6aaaac776987-kube-api-access-hfml5\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.259296 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.259289    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7brm\" (UniqueName: \"kubernetes.io/projected/f1fea5aa-6e82-4768-8f35-fb947216fb9f-kube-api-access-j7brm\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.261590 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.261573    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4sb\" (UniqueName: \"kubernetes.io/projected/cd48639e-7f2c-4383-9716-29c1b4e68994-kube-api-access-pm4sb\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.294921 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.294902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-log-socket\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.295017 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.294930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-k8s-cni-cncf-io\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295017 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.294946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-bin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295017 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.294963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-lib-modules\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.295017 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.294978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-script-lib\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.295017 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.294993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.295017 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-sys\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-kubelet-config\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295035    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-k8s-cni-cncf-io\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295050    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-log-socket\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-multus-certs\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295081    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-bin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-agent-certs\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295109    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-sys\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-ovn\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295137    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-lib-modules\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295149    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-kubelet-config\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295155    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-socket-dir-parent\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295195    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-multus-certs\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-hostroot\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crfc\" (UniqueName: \"kubernetes.io/projected/cfd92d06-4598-48c4-822b-d0dff1df2936-kube-api-access-6crfc\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295197    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-ovn\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-hostroot\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.295324 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295253    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-socket-dir-parent\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-systemd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-slash\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295341    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-systemd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295311    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295361    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-slash\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-cnibin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-cni-binary-copy\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295384    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-config\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295444    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-cnibin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-var-lib-kubelet\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-registration-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8924fe8-5503-4a1d-a361-18221c829534-host-slash\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295579    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8924fe8-5503-4a1d-a361-18221c829534-host-slash\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.296249 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295578    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-registration-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295615    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-var-lib-kubelet\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-kubelet\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-systemd-units\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-netd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295692    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-script-lib\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-env-overrides\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovn-node-metrics-cert\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295751    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-netd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295751    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-systemd-units\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-dbus\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8ch\" (UniqueName: \"kubernetes.io/projected/b8924fe8-5503-4a1d-a361-18221c829534-kube-api-access-cb8ch\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-netns\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8924fe8-5503-4a1d-a361-18221c829534-iptables-alerter-script\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-kubelet\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-etc-kubernetes\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysconfig\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.297320 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295897    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-dbus\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295881    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-kubelet\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-konnectivity-ca\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-daemon-config\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-device-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.295995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbn8\" (UniqueName: \"kubernetes.io/projected/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kube-api-access-crbn8\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-var-lib-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xtl\" (UniqueName: \"kubernetes.io/projected/fca86dc7-9dba-44fa-af4b-209629079b49-kube-api-access-26xtl\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-modprobe-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-run\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-host\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-socket-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-sys-fs\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-etc-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-netns\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.298236 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-multus\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296364    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-multus\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296517    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-config\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-conf\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt5k\" (UniqueName: \"kubernetes.io/projected/48d98419-9d48-4a09-96cd-940bee92d0b3-kube-api-access-2tt5k\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296617    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-conf\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296640    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-cni-binary-copy\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296682    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-env-overrides\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296728    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-netns\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-system-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-systemd\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.296782    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-tuned\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299035 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296832    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-konnectivity-ca\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.296884    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:37:59.796833154 +0000 UTC m=+3.168408242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296922    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-system-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296930    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296964    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-kubelet\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296975    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-etc-kubernetes\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.296882    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297014    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysconfig\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-tmp\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-node-log\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-bin\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-conf-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-kubernetes\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297153    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-systemd\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtft\" (UniqueName: \"kubernetes.io/projected/0808db1b-d15b-47c9-aa62-8a1a6309398b-kube-api-access-xhtft\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297277    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-device-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.299520 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297281    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-socket-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297331    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-daemon-config\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297356    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-modprobe-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297373    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297388    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-var-lib-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297403    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-run\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297430    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-bin\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297438    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-host\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297441    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-conf-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297489    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-node-log\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297501    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.297508    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297518    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-kubernetes\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297548    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-netns\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.297557    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:59.797539523 +0000 UTC m=+3.169114609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297550    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-etc-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297594    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-sys-fs\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.300064 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297791    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-agent-certs\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.300533 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.297974    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8924fe8-5503-4a1d-a361-18221c829534-iptables-alerter-script\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.300533 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.299084    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovn-node-metrics-cert\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.300533 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.299479    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-tuned\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.300533 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.299554    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-tmp\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.360574 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.360518    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:59.360574 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.360538    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:59.360574 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.360552    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:59.360806 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.360617    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:37:59.860592855 +0000 UTC m=+3.232167915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:59.361826 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.361777    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crfc\" (UniqueName: \"kubernetes.io/projected/cfd92d06-4598-48c4-822b-d0dff1df2936-kube-api-access-6crfc\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.361826 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.361803    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8ch\" (UniqueName: \"kubernetes.io/projected/b8924fe8-5503-4a1d-a361-18221c829534-kube-api-access-cb8ch\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.363579 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.363560    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtft\" (UniqueName: \"kubernetes.io/projected/0808db1b-d15b-47c9-aa62-8a1a6309398b-kube-api-access-xhtft\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.363682 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.363607    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt5k\" (UniqueName: \"kubernetes.io/projected/48d98419-9d48-4a09-96cd-940bee92d0b3-kube-api-access-2tt5k\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.363862 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.363843    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbn8\" (UniqueName: \"kubernetes.io/projected/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kube-api-access-crbn8\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.363916 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.363901    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xtl\" (UniqueName: \"kubernetes.io/projected/fca86dc7-9dba-44fa-af4b-209629079b49-kube-api-access-26xtl\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.385334 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.385314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:37:59.395987 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.395957    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fea5aa_6e82_4768_8f35_fb947216fb9f.slice/crio-b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d WatchSource:0}: Error finding container b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d: Status 404 returned error can't find the container with id b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d
Oct 30 18:37:59.403398 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.403380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:37:59.406251 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.406235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4t4x"
Oct 30 18:37:59.411902 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.411886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:37:59.412840 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.412807    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode510aafb_e0d8_4898_a6ac_6aaaac776987.slice/crio-a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0 WatchSource:0}: Error finding container a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0: Status 404 returned error can't find the container with id a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0
Oct 30 18:37:59.413705 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.413686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:37:59.415972 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.415952    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd92d06_4598_48c4_822b_d0dff1df2936.slice/crio-bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f WatchSource:0}: Error finding container bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f: Status 404 returned error can't find the container with id bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f
Oct 30 18:37:59.417121 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.417097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:37:59.424103 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.424086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:37:59.426100 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.426077    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd48639e_7f2c_4383_9716_29c1b4e68994.slice/crio-9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1 WatchSource:0}: Error finding container 9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1: Status 404 returned error can't find the container with id 9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1
Oct 30 18:37:59.426885 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.426854    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d98419_9d48_4a09_96cd_940bee92d0b3.slice/crio-85d068f79b565aa8e1344d5a3a1f24e5dae9654ab848c1daa659018c75c1bcad WatchSource:0}: Error finding container 85d068f79b565aa8e1344d5a3a1f24e5dae9654ab848c1daa659018c75c1bcad: Status 404 returned error can't find the container with id 85d068f79b565aa8e1344d5a3a1f24e5dae9654ab848c1daa659018c75c1bcad
Oct 30 18:37:59.431762 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.431742    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1b72b8_3c48_4426_b83c_94bc84b69bf4.slice/crio-eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5 WatchSource:0}: Error finding container eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5: Status 404 returned error can't find the container with id eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5
Oct 30 18:37:59.435610 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.435568    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8924fe8_5503_4a1d_a361_18221c829534.slice/crio-2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088 WatchSource:0}: Error finding container 2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088: Status 404 returned error can't find the container with id 2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088
Oct 30 18:37:59.435718 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.435703    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:59.449431 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.449404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:37:59.451671 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.451650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:37:59.460724 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.460702    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d87c914_b46f_4c4e_a7d9_3eb4335c463d.slice/crio-eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0 WatchSource:0}: Error finding container eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0: Status 404 returned error can't find the container with id eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0
Oct 30 18:37:59.463473 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:37:59.463453    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0808db1b_d15b_47c9_aa62_8a1a6309398b.slice/crio-1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054 WatchSource:0}: Error finding container 1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054: Status 404 returned error can't find the container with id 1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054
Oct 30 18:37:59.800110 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.800062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:37:59.800321 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.800138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:37:59.800321 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.800275    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:59.800437 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.800332    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:59.800437 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.800375    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:00.800352048 +0000 UTC m=+4.171927111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:59.800437 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.800398    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:00.800386912 +0000 UTC m=+4.171961978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:59.900959 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:37:59.900921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:37:59.901211 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.901186    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:59.901283 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.901216    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:59.901283 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.901234    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:59.901380 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:37:59.901302    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:00.901281892 +0000 UTC m=+4.272856956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:00.136515 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.136432    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fdf4b" event={"ID":"5f1b72b8-3c48-4426-b83c-94bc84b69bf4","Type":"ContainerStarted","Data":"eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5"}
Oct 30 18:38:00.137926 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.137872    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerStarted","Data":"9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1"}
Oct 30 18:38:00.139288 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.139261    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054"}
Oct 30 18:38:00.140625 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.140566    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc" event={"ID":"9d87c914-b46f-4c4e-a7d9-3eb4335c463d","Type":"ContainerStarted","Data":"eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0"}
Oct 30 18:38:00.142210 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.142172    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rvg7d" event={"ID":"b8924fe8-5503-4a1d-a361-18221c829534","Type":"ContainerStarted","Data":"2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088"}
Oct 30 18:38:00.143883 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.143859    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4md92" event={"ID":"48d98419-9d48-4a09-96cd-940bee92d0b3","Type":"ContainerStarted","Data":"85d068f79b565aa8e1344d5a3a1f24e5dae9654ab848c1daa659018c75c1bcad"}
Oct 30 18:38:00.145403 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.145373    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4t4x" event={"ID":"cfd92d06-4598-48c4-822b-d0dff1df2936","Type":"ContainerStarted","Data":"bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f"}
Oct 30 18:38:00.147196 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.147167    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7vv9s" event={"ID":"e510aafb-e0d8-4898-a6ac-6aaaac776987","Type":"ContainerStarted","Data":"a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0"}
Oct 30 18:38:00.148308 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.148285    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4ps2" event={"ID":"f1fea5aa-6e82-4768-8f35-fb947216fb9f","Type":"ContainerStarted","Data":"b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d"}
Oct 30 18:38:00.186974 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.186910    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:59 +0000 UTC" deadline="2028-06-26 04:09:06.949139117 +0000 UTC"
Oct 30 18:38:00.186974 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.186971    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14505h31m6.762171898s"
Oct 30 18:38:00.811140 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.811050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:00.811140 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.811127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:00.811373 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.811331    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:00.811428 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.811399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:02.811375979 +0000 UTC m=+6.182951046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:00.811495 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.811468    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:00.811549 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.811504    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:02.811491173 +0000 UTC m=+6.183066242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:00.912586 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:00.912409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:00.912586 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.912582    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:00.912868 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.912604    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:00.912868 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.912617    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:00.912868 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:00.912703    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:02.912683282 +0000 UTC m=+6.284258348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:01.128880 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:01.128793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:01.129070 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:01.128935    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:01.129070 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:01.129042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:01.129261 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:01.129138    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:01.129261 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:01.129219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:01.129391 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:01.129320    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:01.187715 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:01.187667    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:59 +0000 UTC" deadline="2028-04-02 22:22:23.054260305 +0000 UTC"
Oct 30 18:38:01.187715 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:01.187711    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12483h44m21.866554759s"
Oct 30 18:38:02.834288 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:02.834246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:02.834790 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:02.834324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:02.834790 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.834457    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:02.834790 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.834521    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:06.834503304 +0000 UTC m=+10.206078370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:02.834790 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.834592    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:02.834790 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.834626    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:06.834613929 +0000 UTC m=+10.206188994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:02.935565 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:02.935524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:02.935765 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.935731    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:02.935765 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.935756    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:02.935878 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.935770    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:02.935878 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:02.935836    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:06.935814311 +0000 UTC m=+10.307389417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:03.132490 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.128438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:03.132490 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:03.128583    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:03.132490 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.128685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:03.132490 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:03.128771    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:03.132490 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.128830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:03.132490 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:03.128892    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:03.183190 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.183151    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"c6d9ef131d394299abae730c3d467d74","Type":"ContainerStarted","Data":"3e2047a7fd4434e1f7b0a711acf83a078f27f4331b0b7a7863395e42afa5d2a5"}
Oct 30 18:38:03.186643 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.186614    2580 generic.go:356] "Generic (PLEG): container finished" podID="d47490914e2d482394d4603a5e95f123" containerID="f405a5e8a4c94710d5ff91502998311e9598fa2010745d2e9bf5b46487ff8469" exitCode=0
Oct 30 18:38:03.186774 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.186722    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"d47490914e2d482394d4603a5e95f123","Type":"ContainerDied","Data":"f405a5e8a4c94710d5ff91502998311e9598fa2010745d2e9bf5b46487ff8469"}
Oct 30 18:38:03.204275 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.203725    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerStarted","Data":"eaa9e681a52fcee43eb53da434c4c79d525c899eede6c276bb77736c67b1492a"}
Oct 30 18:38:03.430998 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:03.430835    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal" podStartSLOduration=4.430814282 podStartE2EDuration="4.430814282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:03.306116584 +0000 UTC m=+6.677691670" watchObservedRunningTime="2026-10-30 18:38:03.430814282 +0000 UTC m=+6.802389371"
Oct 30 18:38:04.219484 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:04.219299    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="eaa9e681a52fcee43eb53da434c4c79d525c899eede6c276bb77736c67b1492a" exitCode=0
Oct 30 18:38:04.219484 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:04.219413    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerDied","Data":"eaa9e681a52fcee43eb53da434c4c79d525c899eede6c276bb77736c67b1492a"}
Oct 30 18:38:04.224298 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:04.224229    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"d47490914e2d482394d4603a5e95f123","Type":"ContainerStarted","Data":"26441fee0587bab0871583a948d68b2bc25417b7036fc98ed8a727ce98dbe0fa"}
Oct 30 18:38:05.136798 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:05.136765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:05.136999 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:05.136922    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:05.137076 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:05.137009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:05.137136 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:05.137091    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:05.137192 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:05.137145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:05.137252 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:05.137208    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:05.232598 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:05.232546    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fdf4b" event={"ID":"5f1b72b8-3c48-4426-b83c-94bc84b69bf4","Type":"ContainerStarted","Data":"d2b23b3892898c06b7a8e5809458c74da798f7fabdd36c545e4260f151069b81"}
Oct 30 18:38:05.355971 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:05.355907    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" podStartSLOduration=6.3558866720000005 podStartE2EDuration="6.355886672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:04.347348733 +0000 UTC m=+7.718923818" watchObservedRunningTime="2026-10-30 18:38:05.355886672 +0000 UTC m=+8.727461761"
Oct 30 18:38:06.240843 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:06.240804    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7vv9s" event={"ID":"e510aafb-e0d8-4898-a6ac-6aaaac776987","Type":"ContainerStarted","Data":"897a5bf0faf78917f1b3c9e96d80cfbab30aa4170027013e33697d2ce8983a23"}
Oct 30 18:38:06.354131 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:06.353904    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fdf4b" podStartSLOduration=3.639114598 podStartE2EDuration="8.353882595s" totalImagesPullingTime="4.714767997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:05.35704831 +0000 UTC m=+8.728623393" watchObservedRunningTime="2026-10-30 18:38:06.353882595 +0000 UTC m=+9.725457677"
Oct 30 18:38:06.869406 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:06.869365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:06.869606 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:06.869431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:06.869606 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.869545    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:06.869728 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.869622    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:14.869603764 +0000 UTC m=+18.241178837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:06.869728 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.869695    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:06.869845 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.869731    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:14.869720717 +0000 UTC m=+18.241295782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:06.970789 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:06.970405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:06.970789 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.970661    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:06.970789 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.970688    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:06.970789 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.970700    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:06.970789 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:06.970759    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:14.97074143 +0000 UTC m=+18.342316506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:07.133355 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:07.132957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:07.133355 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:07.133076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:07.133355 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:07.133138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:07.133355 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:07.133220    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:07.133355 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:07.133260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:07.133355 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:07.133324    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:07.244923 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:07.244891    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc" event={"ID":"9d87c914-b46f-4c4e-a7d9-3eb4335c463d","Type":"ContainerStarted","Data":"294fced60c37cc08c911b78672d8c956ade58d17d98d382aee4c016390be2b36"}
Oct 30 18:38:08.818284 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:08.818067    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:38:09.063298 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.063187    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:38:08.818283159Z","UUID":"e8356992-377a-4d73-b256-937a838853f0","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:38:09.064851 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.064828    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:38:09.064961 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.064860    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:38:09.128292 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.128259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:09.128433 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.128332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:09.128433 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.128334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:09.128433 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:09.128413    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:09.128623 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:09.128523    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:09.128686 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:09.128669    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:09.251163 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.251128    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="2333721f3f1cba4c94883f24dfb5c0b8e8f71fc944833151154a5ac3b0698cc9" exitCode=0
Oct 30 18:38:09.251333 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.251223    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerDied","Data":"2333721f3f1cba4c94883f24dfb5c0b8e8f71fc944833151154a5ac3b0698cc9"}
Oct 30 18:38:09.254257 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.254223    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc" event={"ID":"9d87c914-b46f-4c4e-a7d9-3eb4335c463d","Type":"ContainerStarted","Data":"5f255ab7d77c9fac294cbdf750a2a89a53eec164c7fe9cc1506e6dfc96780789"}
Oct 30 18:38:09.256663 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.256639    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4t4x" event={"ID":"cfd92d06-4598-48c4-822b-d0dff1df2936","Type":"ContainerStarted","Data":"a7b5d453ab797ea4124b3838dc8d09c6e8b8cfd94eebcc297a6a0a25cf496e49"}
Oct 30 18:38:09.258259 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.258236    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4ps2" event={"ID":"f1fea5aa-6e82-4768-8f35-fb947216fb9f","Type":"ContainerStarted","Data":"9344a82860a23edcbb6a59d413e21043fb420e59f498fef943e680c8e9fdc36a"}
Oct 30 18:38:09.368788 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.368695    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7vv9s" podStartSLOduration=5.414913825 podStartE2EDuration="11.368680575s" totalImagesPullingTime="5.95376675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:06.353750832 +0000 UTC m=+9.725325917" watchObservedRunningTime="2026-10-30 18:38:09.368680575 +0000 UTC m=+12.740255660"
Oct 30 18:38:09.484826 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.484777    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n4t4x" podStartSLOduration=2.513828845 podStartE2EDuration="11.484736506s" totalImagesPullingTime="8.970907661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:09.484151661 +0000 UTC m=+12.855726743" watchObservedRunningTime="2026-10-30 18:38:09.484736506 +0000 UTC m=+12.856311587"
Oct 30 18:38:09.597186 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.597142    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s4ps2" podStartSLOduration=2.904145073 podStartE2EDuration="11.597125657s" totalImagesPullingTime="8.692980584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:09.596918424 +0000 UTC m=+12.968493509" watchObservedRunningTime="2026-10-30 18:38:09.597125657 +0000 UTC m=+12.968700740"
Oct 30 18:38:09.670006 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.669977    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:38:09.671110 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:09.671084    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:38:10.261854 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:10.261800    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rvg7d" event={"ID":"b8924fe8-5503-4a1d-a361-18221c829534","Type":"ContainerStarted","Data":"02f40a61e470fc08b9a768ae343215b7f4bafc843756c42cb54c165a8c40fbcb"}
Oct 30 18:38:11.128909 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.128880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:11.129086 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.128922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:11.129086 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:11.129011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:11.129156 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:11.129075    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:11.129156 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.129131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:11.129208 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:11.129196    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:11.266671 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.266629    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="f5810a664575fe0a0d78c0e7dc76b23cd052ad9e2a39f0826ab1215c08721017" exitCode=0
Oct 30 18:38:11.267419 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.266706    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerDied","Data":"f5810a664575fe0a0d78c0e7dc76b23cd052ad9e2a39f0826ab1215c08721017"}
Oct 30 18:38:11.269104 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.268998    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc" event={"ID":"9d87c914-b46f-4c4e-a7d9-3eb4335c463d","Type":"ContainerStarted","Data":"b2a80c3eee630e77a48e303c25eeb6dbe418ac364964c097aa656051098b9539"}
Oct 30 18:38:11.270705 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.270680    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4md92" event={"ID":"48d98419-9d48-4a09-96cd-940bee92d0b3","Type":"ContainerStarted","Data":"ffcc5043e521988cd784b436f74e6433f9e05c2e480005e036b0cb44b94c57b5"}
Oct 30 18:38:11.388898 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.388796    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rvg7d" podStartSLOduration=4.578907812 podStartE2EDuration="13.388778686s" totalImagesPullingTime="8.809870874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:10.393217111 +0000 UTC m=+13.764792191" watchObservedRunningTime="2026-10-30 18:38:11.388778686 +0000 UTC m=+14.760353765"
Oct 30 18:38:11.509256 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.509210    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc" podStartSLOduration=2.802508202 podStartE2EDuration="13.509195899s" totalImagesPullingTime="10.706687697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:11.50854829 +0000 UTC m=+14.880123371" watchObservedRunningTime="2026-10-30 18:38:11.509195899 +0000 UTC m=+14.880770980"
Oct 30 18:38:11.627965 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:11.627918    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4md92" podStartSLOduration=2.525486861 podStartE2EDuration="13.627906047s" totalImagesPullingTime="11.102419186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:11.627528143 +0000 UTC m=+14.999103226" watchObservedRunningTime="2026-10-30 18:38:11.627906047 +0000 UTC m=+14.999481163"
Oct 30 18:38:13.129364 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:13.129146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:13.129797 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:13.129167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:13.129797 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:13.129447    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:13.129797 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:13.129513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:13.129797 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:13.129231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:13.129797 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:13.129622    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:13.278754 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:13.278718    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="c4ba4f3c070d8a055649157f1bbfe3a7f7346abdf729d3333a537a2d9c04982b" exitCode=0
Oct 30 18:38:13.278940 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:13.278769    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerDied","Data":"c4ba4f3c070d8a055649157f1bbfe3a7f7346abdf729d3333a537a2d9c04982b"}
Oct 30 18:38:14.929391 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:14.929368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:14.929727 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:14.929411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:14.929727 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:14.929511    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:14.929727 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:14.929514    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:14.929727 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:14.929558    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:30.929544565 +0000 UTC m=+34.301119625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:14.929727 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:14.929571    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:30.929565165 +0000 UTC m=+34.301140223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:15.029837 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.029715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:15.029959 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.029863    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:15.029959 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.029881    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:15.029959 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.029889    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:15.029959 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.029936    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:38:31.029921269 +0000 UTC m=+34.401496359 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:15.128134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.128111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:15.128237 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.128118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:15.128290 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.128242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:15.128290 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.128120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:15.128383 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.128288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:15.128433 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:15.128379    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:15.286598 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.286564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:38:15.287000 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.286962    2580 generic.go:356] "Generic (PLEG): container finished" podID="0808db1b-d15b-47c9-aa62-8a1a6309398b" containerID="966f69808562e64c0d708009508f22e9e2658ddf29afacc9586f8414ed22f915" exitCode=1
Oct 30 18:38:15.287084 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.287008    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"09eb6ccabf5fa058afd4078e87ded040621bdbbfeaaf66bf799374524f527f8c"}
Oct 30 18:38:15.287136 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.287083    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"0f5836a8666b187cf5fd3f557c319312cb74531be16cd4a6883488a5a944abf2"}
Oct 30 18:38:15.287184 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.287133    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"3244f5ef7d5a771261def370bd4c2b6abe2ae36de4c60140b05f5e0b2f5dbb88"}
Oct 30 18:38:15.287215 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.287183    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"f12342b1d842f36b0ac51225150061290d553c1af62b8d1d66675abd2719ad9e"}
Oct 30 18:38:15.287258 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.287230    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerDied","Data":"966f69808562e64c0d708009508f22e9e2658ddf29afacc9586f8414ed22f915"}
Oct 30 18:38:15.287300 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:15.287280    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"b482c99b879f08f83d1fcd4a3e4f385123f3e2664469163b70bbd8acd3f29754"}
Oct 30 18:38:16.291769 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:16.291738    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="46ba7028298c4cd01441e4abbc7d3aa961658d2940c5af9f4f6ee153b74706c5" exitCode=0
Oct 30 18:38:16.292280 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:16.291776    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerDied","Data":"46ba7028298c4cd01441e4abbc7d3aa961658d2940c5af9f4f6ee153b74706c5"}
Oct 30 18:38:17.131611 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.130354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:17.132498 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.132472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:17.132621 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:17.132492    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:17.132621 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:17.132562    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:17.132621 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.132472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:17.132746 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:17.132703    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:17.297202 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.297173    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="6ffacf4bedca948ca2e8844f14a44aad8e90cdb7591b6778f8c2d710247b3fb0" exitCode=0
Oct 30 18:38:17.297729 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.297257    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerDied","Data":"6ffacf4bedca948ca2e8844f14a44aad8e90cdb7591b6778f8c2d710247b3fb0"}
Oct 30 18:38:17.300462 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.300445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:38:17.300887 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:17.300868    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"e214899dfb28222bdc5929b17d62ca21117d9cc5d3dce6dd612a49c9501d3278"}
Oct 30 18:38:18.306307 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:18.306272    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" event={"ID":"cd48639e-7f2c-4383-9716-29c1b4e68994","Type":"ContainerStarted","Data":"67401589d9cf9c42ae3c7803d0d393fbe17bba2b93c2df843d36f726367d141b"}
Oct 30 18:38:18.429896 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:18.429846    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4ljqw" podStartSLOduration=7.533325529 podStartE2EDuration="20.429832659s" totalImagesPullingTime="12.89650713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:18.429498059 +0000 UTC m=+21.801073140" watchObservedRunningTime="2026-10-30 18:38:18.429832659 +0000 UTC m=+21.801407740"
Oct 30 18:38:19.129047 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:19.129019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:19.129047 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:19.129046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:19.129231 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:19.129027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:19.129231 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:19.129122    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:19.129231 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:19.129188    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:19.129317 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:19.129255    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:20.314047 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.313864    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:38:20.314697 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.314407    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"3baab02fc8afaec533d9224ce177ccbf7be1f9d0851a50c2a3c0c57c30eaccbf"}
Oct 30 18:38:20.314837 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.314817    2580 scope.go:122] "RemoveContainer" containerID="966f69808562e64c0d708009508f22e9e2658ddf29afacc9586f8414ed22f915"
Oct 30 18:38:20.314912 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.314857    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:38:20.314972 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.314942    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:38:20.331751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.331729    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:38:20.331846 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.331833    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:38:20.473282 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:20.473252    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:38:21.128491 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.128462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:21.128658 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.128496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:21.128658 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.128463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:21.128658 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:21.128575    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:21.128753 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:21.128664    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:21.128753 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:21.128726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:21.320046 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.320023    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:38:21.320432 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.320362    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" event={"ID":"0808db1b-d15b-47c9-aa62-8a1a6309398b","Type":"ContainerStarted","Data":"c4bf163c59aad49441a1c4fc8094c118c4ab0d3d405dffe97761fa69f8eb03ba"}
Oct 30 18:38:21.451628 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.451574    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-shstn" podStartSLOduration=8.175128329 podStartE2EDuration="23.451562614s" totalImagesPullingTime="15.276434285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:21.451232543 +0000 UTC m=+24.822807625" watchObservedRunningTime="2026-10-30 18:38:21.451562614 +0000 UTC m=+24.823137695"
Oct 30 18:38:21.950255 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.950226    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77qht"]
Oct 30 18:38:21.950401 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.950350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:21.950439 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:21.950426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:21.950761 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.950748    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qd6bc"]
Oct 30 18:38:21.950830 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.950820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:21.950900 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:21.950887    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:21.951221 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.951201    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdb7t"]
Oct 30 18:38:21.951301 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:21.951288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:21.951387 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:21.951368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:23.128445 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:23.128419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:23.128954 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:23.128418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:23.128954 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:23.128537    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:23.128954 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:23.128603    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:24.107273 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:24.107243    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:38:24.107451 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:24.107397    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:38:24.107829 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:24.107809    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:38:24.128313 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:24.128283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:24.128402 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:24.128385    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:25.128962 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:25.128927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:25.129334 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:25.129029    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:25.129334 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:25.129118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:25.129334 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:25.129203    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:26.129173 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:26.128999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:26.129659 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:26.129277    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:27.129903 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:27.129875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:27.130286 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:27.129953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:27.130286 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:27.129952    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:27.130286 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:27.130032    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:28.128982 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:28.128955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:28.129153 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:28.129055    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:29.129080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:29.129047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:29.129080 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:29.129066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:29.129640 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:29.129174    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:29.129640 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:29.129305    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:30.128233 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:30.128207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:30.128383 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:30.128306    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:30.947331 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:30.947300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:30.947819 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:30.947360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:30.947819 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:30.947449    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:30.947819 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:30.947465    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:30.947819 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:30.947507    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs podName:fca86dc7-9dba-44fa-af4b-209629079b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:39:02.947492081 +0000 UTC m=+66.319067160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs") pod "network-metrics-daemon-qd6bc" (UID: "fca86dc7-9dba-44fa-af4b-209629079b49") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:30.947819 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:30.947523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret podName:8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b nodeName:}" failed. No retries permitted until 2026-10-30 18:39:02.947515894 +0000 UTC m=+66.319090953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret") pod "global-pull-secret-syncer-77qht" (UID: "8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:31.048283 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:31.048254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:31.048431 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:31.048379    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:31.048431 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:31.048399    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:31.048431 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:31.048411    2580 projected.go:196] Error preparing data for projected volume kube-api-access-jn4cw for pod openshift-network-diagnostics/network-check-target-mdb7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:31.048559 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:31.048467    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw podName:9a9fb057-17de-4446-a7cf-0730aaa6c46b nodeName:}" failed. No retries permitted until 2026-10-30 18:39:03.048451002 +0000 UTC m=+66.420026076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jn4cw" (UniqueName: "kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw") pod "network-check-target-mdb7t" (UID: "9a9fb057-17de-4446-a7cf-0730aaa6c46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:31.128060 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:31.128033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:31.128177 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:31.128033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:31.128177 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:31.128129    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:31.128268 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:31.128206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:32.053725 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:32.053695    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4ps2_f1fea5aa-6e82-4768-8f35-fb947216fb9f/dns-node-resolver/0.log"
Oct 30 18:38:32.128569 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:32.128541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:32.128735 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:32.128682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:32.673542 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:32.673508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7vv9s_e510aafb-e0d8-4898-a6ac-6aaaac776987/node-ca/0.log"
Oct 30 18:38:33.131046 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:33.131017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:33.131046 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:33.131049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:33.131552 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:33.131207    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:33.131552 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:33.131250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:33.131552 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:33.131376    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:33.131552 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:33.131452    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:35.131457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:35.131425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:35.131457 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:35.131446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:35.131953 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:35.131425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:35.131953 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:35.131525    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:35.131953 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:35.131665    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:35.131953 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:35.131701    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:37.129343 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:37.129315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:37.129775 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:37.129469    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:37.129775 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:37.129496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:37.129775 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:37.129526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:37.129775 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:37.129612    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:37.129775 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:37.129668    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:39.129069 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:39.129037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:39.129069 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:39.129055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:39.129577 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:39.129055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:39.129577 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:39.129161    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:39.129577 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:39.129218    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:39.129577 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:39.129305    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:41.131476 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:41.131445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:41.131954 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:41.131557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:41.131954 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:41.131598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:41.131954 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:41.131571    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdb7t" podUID="9a9fb057-17de-4446-a7cf-0730aaa6c46b"
Oct 30 18:38:41.131954 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:41.131678    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77qht" podUID="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b"
Oct 30 18:38:41.131954 ip-10-0-134-133 kubenswrapper[2580]: E1030 18:38:41.131778    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd6bc" podUID="fca86dc7-9dba-44fa-af4b-209629079b49"
Oct 30 18:38:42.309807 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.309782    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:38:42.310152 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.309886    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:38:42.440208 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.440182    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-55t9g"]
Oct 30 18:38:42.466342 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.466317    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pk298"]
Oct 30 18:38:42.466466 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.466447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.482401 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.482380    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kz4df"]
Oct 30 18:38:42.482524 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.482509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.502797 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.502775    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55t9g"]
Oct 30 18:38:42.502893 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.502802    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pk298"]
Oct 30 18:38:42.502893 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.502822    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kz4df"]
Oct 30 18:38:42.502953 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.502894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.520166 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.520148    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:42.520268 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.520208    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:42.520350 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.520336    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-64dnq\""
Oct 30 18:38:42.520404 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.520364    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:42.535733 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.535711    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:42.535814 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.535718    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvdpg\""
Oct 30 18:38:42.543424 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.543404    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:42.543528 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.543517    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:42.557512 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.557495    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:42.620686 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.620670    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:42.620813 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.620800    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlkjn\""
Oct 30 18:38:42.620880 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.620869    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:42.629027 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47825bca-e2dc-480b-b29e-3e071b260fd9-config-volume\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.629074 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkhk\" (UniqueName: \"kubernetes.io/projected/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-api-access-7rkhk\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.629074 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwsx\" (UniqueName: \"kubernetes.io/projected/47825bca-e2dc-480b-b29e-3e071b260fd9-kube-api-access-5dwsx\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.629138 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57299938-b4d9-45b6-9650-e0f0c9b50c80-data-volume\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.629168 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.629264 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47825bca-e2dc-480b-b29e-3e071b260fd9-metrics-tls\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.629297 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4gz\" (UniqueName: \"kubernetes.io/projected/cc0a19a4-6a53-4252-8deb-30327d884859-kube-api-access-pm4gz\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.629370 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57299938-b4d9-45b6-9650-e0f0c9b50c80-crio-socket\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.629402 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57299938-b4d9-45b6-9650-e0f0c9b50c80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.629447 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0a19a4-6a53-4252-8deb-30327d884859-cert\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.629480 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.629461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47825bca-e2dc-480b-b29e-3e071b260fd9-tmp-dir\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.730410 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47825bca-e2dc-480b-b29e-3e071b260fd9-tmp-dir\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.730410 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47825bca-e2dc-480b-b29e-3e071b260fd9-config-volume\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.730410 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkhk\" (UniqueName: \"kubernetes.io/projected/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-api-access-7rkhk\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.730604 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwsx\" (UniqueName: \"kubernetes.io/projected/47825bca-e2dc-480b-b29e-3e071b260fd9-kube-api-access-5dwsx\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.730604 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57299938-b4d9-45b6-9650-e0f0c9b50c80-data-volume\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.730604 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.730604 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47825bca-e2dc-480b-b29e-3e071b260fd9-metrics-tls\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.730800 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4gz\" (UniqueName: \"kubernetes.io/projected/cc0a19a4-6a53-4252-8deb-30327d884859-kube-api-access-pm4gz\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.730800 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57299938-b4d9-45b6-9650-e0f0c9b50c80-crio-socket\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.730800 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57299938-b4d9-45b6-9650-e0f0c9b50c80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.730800 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0a19a4-6a53-4252-8deb-30327d884859-cert\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.730975 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730813    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47825bca-e2dc-480b-b29e-3e071b260fd9-tmp-dir\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.730975 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730863    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57299938-b4d9-45b6-9650-e0f0c9b50c80-data-volume\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.730975 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.730891    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57299938-b4d9-45b6-9650-e0f0c9b50c80-crio-socket\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.731123 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.731107    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47825bca-e2dc-480b-b29e-3e071b260fd9-config-volume\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.731354 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.731331    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.734527 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.734501    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47825bca-e2dc-480b-b29e-3e071b260fd9-metrics-tls\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.734624 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.734501    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57299938-b4d9-45b6-9650-e0f0c9b50c80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.734708 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.734693    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0a19a4-6a53-4252-8deb-30327d884859-cert\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.792016 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.791990    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwsx\" (UniqueName: \"kubernetes.io/projected/47825bca-e2dc-480b-b29e-3e071b260fd9-kube-api-access-5dwsx\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:42.792119 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.792099    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkhk\" (UniqueName: \"kubernetes.io/projected/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-api-access-7rkhk\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.792192 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.792177    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4gz\" (UniqueName: \"kubernetes.io/projected/cc0a19a4-6a53-4252-8deb-30327d884859-kube-api-access-pm4gz\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:42.832345 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.832322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:38:42.860646 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:42.860630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:43.088936 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.088875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:38:43.132372 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.132348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:38:43.132372 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.132371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:38:43.132564 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.132491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:38:43.186309 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186286    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:43.186573 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186554    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l87lm\""
Oct 30 18:38:43.186573 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186593    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:43.186751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186623    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swx9z\""
Oct 30 18:38:43.186751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186636    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:43.186751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186643    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:43.186751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.186728    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dxlbz\""
Oct 30 18:38:43.338971 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.338929    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pk298"]
Oct 30 18:38:43.341601 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:43.341565    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57299938_b4d9_45b6_9650_e0f0c9b50c80.slice/crio-5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9 WatchSource:0}: Error finding container 5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9: Status 404 returned error can't find the container with id 5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9
Oct 30 18:38:43.341796 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.341736    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kz4df"]
Oct 30 18:38:43.341916 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:43.341893    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47825bca_e2dc_480b_b29e_3e071b260fd9.slice/crio-0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845 WatchSource:0}: Error finding container 0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845: Status 404 returned error can't find the container with id 0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845
Oct 30 18:38:43.365336 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.365309    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pk298" event={"ID":"57299938-b4d9-45b6-9650-e0f0c9b50c80","Type":"ContainerStarted","Data":"5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9"}
Oct 30 18:38:43.367150 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.367123    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kz4df" event={"ID":"47825bca-e2dc-480b-b29e-3e071b260fd9","Type":"ContainerStarted","Data":"0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845"}
Oct 30 18:38:43.459734 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:43.459711    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0a19a4_6a53_4252_8deb_30327d884859.slice/crio-d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb WatchSource:0}: Error finding container d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb: Status 404 returned error can't find the container with id d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb
Oct 30 18:38:43.460171 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:43.460150    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55t9g"]
Oct 30 18:38:44.370956 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:44.370923    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55t9g" event={"ID":"cc0a19a4-6a53-4252-8deb-30327d884859","Type":"ContainerStarted","Data":"d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb"}
Oct 30 18:38:44.372713 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:44.372681    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pk298" event={"ID":"57299938-b4d9-45b6-9650-e0f0c9b50c80","Type":"ContainerStarted","Data":"73911c8b6cc3ab332cdd589ddad481d6ec2060cb681e0cd5ffca61f66fae086e"}
Oct 30 18:38:45.377557 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:45.377464    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pk298" event={"ID":"57299938-b4d9-45b6-9650-e0f0c9b50c80","Type":"ContainerStarted","Data":"b1dd82d3a23f5ff2d2473d16e79dc4c8ce9384cc1febc36ad162d964674bc7ff"}
Oct 30 18:38:46.382300 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:46.382259    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kz4df" event={"ID":"47825bca-e2dc-480b-b29e-3e071b260fd9","Type":"ContainerStarted","Data":"5c7a8eab0c9d20c4c0eb8938c7713889a978c06a53bbf5e6b6f22b6749635029"}
Oct 30 18:38:46.382777 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:46.382324    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kz4df" event={"ID":"47825bca-e2dc-480b-b29e-3e071b260fd9","Type":"ContainerStarted","Data":"212f08f71e8cc283c2f1c818c269621a456a271250b9084407b036969a70724a"}
Oct 30 18:38:46.382777 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:46.382397    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:46.501027 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:46.500966    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kz4df" podStartSLOduration=1.993562495 podStartE2EDuration="4.500947487s" totalImagesPullingTime="2.507384992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:46.500653918 +0000 UTC m=+49.872229000" watchObservedRunningTime="2026-10-30 18:38:46.500947487 +0000 UTC m=+49.872522571"
Oct 30 18:38:47.386838 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.386812    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pk298" event={"ID":"57299938-b4d9-45b6-9650-e0f0c9b50c80","Type":"ContainerStarted","Data":"0a7e3e66c895c67aea549d6e5a910ddecb14e8b34297f31b759b78e0d5bd1ade"}
Oct 30 18:38:47.388216 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.388197    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55t9g" event={"ID":"cc0a19a4-6a53-4252-8deb-30327d884859","Type":"ContainerStarted","Data":"6d57f6c6b06c7e11cda9a01378984112783bbbb7425ad8f814318d8f00e42103"}
Oct 30 18:38:47.609230 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.609165    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"]
Oct 30 18:38:47.610879 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.610866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.666975 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.666946    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 18:38:47.667121 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.667014    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:38:47.667121 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.667066    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:38:47.667268 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.667143    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4mf4w\""
Oct 30 18:38:47.667268 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.667224    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:38:47.667268 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.667225    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 18:38:47.668098 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.668076    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"]
Oct 30 18:38:47.767202 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.767179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.767202 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.767205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9f7t\" (UniqueName: \"kubernetes.io/projected/522412d0-7818-4a3d-975f-c816eb5d7a13-kube-api-access-b9f7t\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.767363 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.767232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.767401 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.767368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/522412d0-7818-4a3d-975f-c816eb5d7a13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.867813 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.867750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.867813 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.867775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9f7t\" (UniqueName: \"kubernetes.io/projected/522412d0-7818-4a3d-975f-c816eb5d7a13-kube-api-access-b9f7t\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.867813 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.867802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.868033 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.867999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/522412d0-7818-4a3d-975f-c816eb5d7a13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.868621 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.868537    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/522412d0-7818-4a3d-975f-c816eb5d7a13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.870269 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.870247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.870347 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.870322    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:47.929218 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:47.929186    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9f7t\" (UniqueName: \"kubernetes.io/projected/522412d0-7818-4a3d-975f-c816eb5d7a13-kube-api-access-b9f7t\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:48.221198 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:48.221175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:38:48.535214 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:48.535124    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-55t9g" podStartSLOduration=3.469031443 podStartE2EDuration="6.535106122s" totalImagesPullingTime="3.066074679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:48.53473142 +0000 UTC m=+51.906306501" watchObservedRunningTime="2026-10-30 18:38:48.535106122 +0000 UTC m=+51.906681203"
Oct 30 18:38:48.585137 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:48.585109    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522412d0_7818_4a3d_975f_c816eb5d7a13.slice/crio-c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82 WatchSource:0}: Error finding container c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82: Status 404 returned error can't find the container with id c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82
Oct 30 18:38:48.585487 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:48.585467    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"]
Oct 30 18:38:48.653369 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:48.653326    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pk298" podStartSLOduration=2.884180199 podStartE2EDuration="6.653313904s" totalImagesPullingTime="3.769133705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:48.652688993 +0000 UTC m=+52.024264073" watchObservedRunningTime="2026-10-30 18:38:48.653313904 +0000 UTC m=+52.024888984"
Oct 30 18:38:49.395835 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:49.395799    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m" event={"ID":"522412d0-7818-4a3d-975f-c816eb5d7a13","Type":"ContainerStarted","Data":"c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82"}
Oct 30 18:38:50.400468 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:50.400440    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m" event={"ID":"522412d0-7818-4a3d-975f-c816eb5d7a13","Type":"ContainerStarted","Data":"297a10af268ff80964db419dd41f255c4c8815aa3ecf5058bb779cd8591760df"}
Oct 30 18:38:51.405474 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:51.405434    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m" event={"ID":"522412d0-7818-4a3d-975f-c816eb5d7a13","Type":"ContainerStarted","Data":"73aec5e82758e0f6b4101399d7ef464e4616fb1b40e864004e9779bd7705c5ee"}
Oct 30 18:38:51.524814 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:51.524766    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m" podStartSLOduration=2.805261315 podStartE2EDuration="4.524753449s" totalImagesPullingTime="1.719492134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:51.523903385 +0000 UTC m=+54.895478465" watchObservedRunningTime="2026-10-30 18:38:51.524753449 +0000 UTC m=+54.896328529"
Oct 30 18:38:52.334614 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:52.334574    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:38:54.201641 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.201607    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"]
Oct 30 18:38:54.203662 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.203648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.256907 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.256885    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:38:54.307918 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.307895    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jxx4c\""
Oct 30 18:38:54.308062 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.307982    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 18:38:54.309048 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.309028    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"]
Oct 30 18:38:54.317577 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.317554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kk45\" (UniqueName: \"kubernetes.io/projected/cf611fab-4203-425e-8db6-6ab6072ec4c2-kube-api-access-5kk45\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.317706 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.317688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.317745 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.317716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.317787 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.317776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf611fab-4203-425e-8db6-6ab6072ec4c2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.338272 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.338246    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-49tz6"]
Oct 30 18:38:54.340349 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.340336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.393271 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.393250    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w6qhz\""
Oct 30 18:38:54.414238 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.414210    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:38:54.414238 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.414209    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:38:54.418641 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.418622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk45\" (UniqueName: \"kubernetes.io/projected/cf611fab-4203-425e-8db6-6ab6072ec4c2-kube-api-access-5kk45\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.418745 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.418722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.418792 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.418749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.418792 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.418776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf611fab-4203-425e-8db6-6ab6072ec4c2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.419436 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.419415    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf611fab-4203-425e-8db6-6ab6072ec4c2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.421365 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.421344    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.421437 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.421411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.434124 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.434101    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"]
Oct 30 18:38:54.436119 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.436106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.481002 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.480944    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk45\" (UniqueName: \"kubernetes.io/projected/cf611fab-4203-425e-8db6-6ab6072ec4c2-kube-api-access-5kk45\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.491994 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.491974    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:38:54.492629 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.492610    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:38:54.492725 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.492711    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wrflg\""
Oct 30 18:38:54.492949 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.492932    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 18:38:54.494301 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.494285    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"]
Oct 30 18:38:54.515102 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.515078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:38:54.519047 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-wtmp\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519095 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519095 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-textfile\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519156 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-root\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519194 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nwv\" (UniqueName: \"kubernetes.io/projected/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-kube-api-access-69nwv\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519239 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-sys\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519277 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-tls\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519374 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-metrics-client-ca\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.519438 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.519390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-accelerators-collector-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.527679 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.527655    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 18:38:54.620518 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrcr\" (UniqueName: \"kubernetes.io/projected/549292be-cfde-49a0-bd85-b1854a005bff-kube-api-access-vwrcr\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.620690 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/549292be-cfde-49a0-bd85-b1854a005bff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.620690 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.620779 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-accelerators-collector-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.620779 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-wtmp\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nwv\" (UniqueName: \"kubernetes.io/projected/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-kube-api-access-69nwv\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-root\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-sys\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-tls\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.620970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-metrics-client-ca\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621001    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-root\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-textfile\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.621134 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.621759 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621140    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-wtmp\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621759 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621535    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-textfile\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621759 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621649    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-accelerators-collector-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.621759 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.621654    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-sys\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.622053 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.622036    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-metrics-client-ca\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.623950 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.623929    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.624145 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.624128    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-tls\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.682804 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.682780    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nwv\" (UniqueName: \"kubernetes.io/projected/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-kube-api-access-69nwv\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.722456 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.722430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.722612 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.722467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.722612 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.722540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrcr\" (UniqueName: \"kubernetes.io/projected/549292be-cfde-49a0-bd85-b1854a005bff-kube-api-access-vwrcr\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.722612 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.722600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/549292be-cfde-49a0-bd85-b1854a005bff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.722766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.722623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.722766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.722671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.723043 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.723009    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/549292be-cfde-49a0-bd85-b1854a005bff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.723306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.723295    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.723383 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.723297    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.724896 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.724876    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.724987 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.724950    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.784835 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.784769    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrcr\" (UniqueName: \"kubernetes.io/projected/549292be-cfde-49a0-bd85-b1854a005bff-kube-api-access-vwrcr\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:54.926264 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.926240    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"]
Oct 30 18:38:54.926375 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:54.926237    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf611fab_4203_425e_8db6_6ab6072ec4c2.slice/crio-87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d WatchSource:0}: Error finding container 87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d: Status 404 returned error can't find the container with id 87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d
Oct 30 18:38:54.951671 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:54.951646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:38:54.960172 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:54.960140    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbb88e7_35a1_4d40_9365_66a6ba00adb9.slice/crio-4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b WatchSource:0}: Error finding container 4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b: Status 404 returned error can't find the container with id 4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b
Oct 30 18:38:55.047982 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:55.047924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:38:55.419105 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:55.419015    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl" event={"ID":"cf611fab-4203-425e-8db6-6ab6072ec4c2","Type":"ContainerStarted","Data":"06810407a8303f786b91144f9ccd81450fbd176f5332d7b041803acc7cf14033"}
Oct 30 18:38:55.419105 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:55.419075    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl" event={"ID":"cf611fab-4203-425e-8db6-6ab6072ec4c2","Type":"ContainerStarted","Data":"828a45e8e6837d1da5ba70fe11f5881e8c6738e69cbdf9e8084ba24197e92c40"}
Oct 30 18:38:55.419540 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:55.419108    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl" event={"ID":"cf611fab-4203-425e-8db6-6ab6072ec4c2","Type":"ContainerStarted","Data":"87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d"}
Oct 30 18:38:55.420112 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:55.420084    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49tz6" event={"ID":"bfbb88e7-35a1-4d40-9365-66a6ba00adb9","Type":"ContainerStarted","Data":"4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b"}
Oct 30 18:38:55.461972 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:55.461936    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549292be_cfde_49a0_bd85_b1854a005bff.slice/crio-125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6 WatchSource:0}: Error finding container 125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6: Status 404 returned error can't find the container with id 125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6
Oct 30 18:38:55.462675 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:55.462652    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"]
Oct 30 18:38:56.390358 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.390331    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kz4df"
Oct 30 18:38:56.425603 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.425545    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch" event={"ID":"549292be-cfde-49a0-bd85-b1854a005bff","Type":"ContainerStarted","Data":"125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6"}
Oct 30 18:38:56.680181 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.680157    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:56.683097 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.683078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.742662 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.741416    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhcv2\""
Oct 30 18:38:56.742662 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.741770    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:38:56.742662 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.741820    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:38:56.742662 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.741900    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:38:56.742662 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.741957    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:38:56.757013 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.756987    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:38:56.757196 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.756987    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:38:56.757196 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.757021    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:38:56.757315 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.757225    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:38:56.808731 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.808697    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:38:56.809822 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.809804    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:56.838663 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838751 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838882 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838916 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838952 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.838980 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.839010 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.838977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.839038 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.839029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.839071 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.839050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.839141 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.839114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940543 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940717 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940861 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940861 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940861 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940861 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.940861 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.941077 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.941077 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.941077 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.941077 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.940996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.941376 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.941354    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.942306 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.942280    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.942733 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.942668    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.945720 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.945065    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.945720 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.945235    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.945720 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.945461    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.945720 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.945517    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.945720 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.945674    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.945720 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.945682    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.946077 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.946003    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.946432 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.946408    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:56.946812 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:56.946792    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:57.005639 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.005614    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:57.012526 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.012503    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c948665c7-79g5z"]
Oct 30 18:38:57.016066 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.016044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.070627 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.070560    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 18:38:57.071189 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.070751    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 18:38:57.071189 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.070988    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 18:38:57.071189 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.071064    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 18:38:57.071743 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.071722    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 18:38:57.071862 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.071819    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gkr95\""
Oct 30 18:38:57.071968 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.071941    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1bcilccoormb6\""
Oct 30 18:38:57.075237 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.075219    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c948665c7-79g5z"]
Oct 30 18:38:57.087536 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.087517    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhcv2\""
Oct 30 18:38:57.096804 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.096782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:57.143096 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.142934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143096 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.142970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-grpc-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143096 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.142998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143096 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.143076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143346 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.143228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-metrics-client-ca\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143346 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.143304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrhq\" (UniqueName: \"kubernetes.io/projected/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-kube-api-access-xwrhq\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143439 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.143375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143439 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.143410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-tmp\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.143542 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.143524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.243932 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.243903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244021 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.243963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244021 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.243998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-grpc-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244096 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244141 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244141 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-metrics-client-ca\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244233 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrhq\" (UniqueName: \"kubernetes.io/projected/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-kube-api-access-xwrhq\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244233 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244767 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-tmp\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.244996 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.244971    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-tmp\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.249685 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.245956    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-metrics-client-ca\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.249685 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.247512    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.249685 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.247951    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.249685 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.248894    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.250129 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.250098    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.251181 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.251163    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.251810 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.251793    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-grpc-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.306415 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.306389    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrhq\" (UniqueName: \"kubernetes.io/projected/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-kube-api-access-xwrhq\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.332028 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.332008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:38:57.433194 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.433155    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl" event={"ID":"cf611fab-4203-425e-8db6-6ab6072ec4c2","Type":"ContainerStarted","Data":"fde6b8e59693eabb246255edc40fef14a57d57003efffd706234c2603b0bff34"}
Oct 30 18:38:57.435484 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.435460    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch" event={"ID":"549292be-cfde-49a0-bd85-b1854a005bff","Type":"ContainerStarted","Data":"50d3b3bfab83c4e64c124d448d5b08022c283ae5cdecb58e816cd94b4917b927"}
Oct 30 18:38:57.435597 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.435515    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch" event={"ID":"549292be-cfde-49a0-bd85-b1854a005bff","Type":"ContainerStarted","Data":"9b7c31e057e9f6b9f4d79578274daf206c7db32d53a7af17c59e57a4f5f6352c"}
Oct 30 18:38:57.435597 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.435544    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch" event={"ID":"549292be-cfde-49a0-bd85-b1854a005bff","Type":"ContainerStarted","Data":"0a4fb509f06d7c3fc0d6ad4f488c2881433e2d6b2db17b9c9ddeecf33c26ba38"}
Oct 30 18:38:57.437013 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.436991    2580 generic.go:356] "Generic (PLEG): container finished" podID="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" containerID="5fac431e9713b1bde31eb0800d333edbd22e9fc4e3b34b361c1e7ce628c7ef25" exitCode=0
Oct 30 18:38:57.437119 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.437022    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49tz6" event={"ID":"bfbb88e7-35a1-4d40-9365-66a6ba00adb9","Type":"ContainerDied","Data":"5fac431e9713b1bde31eb0800d333edbd22e9fc4e3b34b361c1e7ce628c7ef25"}
Oct 30 18:38:57.528563 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:57.528536    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01662703_d552_4ae9_bc54_9e7da3c21bf3.slice/crio-56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb WatchSource:0}: Error finding container 56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb: Status 404 returned error can't find the container with id 56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb
Oct 30 18:38:57.530728 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.530708    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:57.552253 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.552150    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl" podStartSLOduration=1.8977093969999999 podStartE2EDuration="3.552135846s" totalImagesPullingTime="1.654426449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:57.551783065 +0000 UTC m=+60.923358147" watchObservedRunningTime="2026-10-30 18:38:57.552135846 +0000 UTC m=+60.923710928"
Oct 30 18:38:57.708732 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.708703    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c948665c7-79g5z"]
Oct 30 18:38:57.708879 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:38:57.708791    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e30ed0_eb80_40bc_87bb_1afbdda8f1e3.slice/crio-b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64 WatchSource:0}: Error finding container b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64: Status 404 returned error can't find the container with id b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64
Oct 30 18:38:57.792755 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:57.792678    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch" podStartSLOduration=2.027447524 podStartE2EDuration="3.79266429s" totalImagesPullingTime="1.765216766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:57.791947024 +0000 UTC m=+61.163522105" watchObservedRunningTime="2026-10-30 18:38:57.79266429 +0000 UTC m=+61.164239371"
Oct 30 18:38:58.442907 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:58.442871    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49tz6" event={"ID":"bfbb88e7-35a1-4d40-9365-66a6ba00adb9","Type":"ContainerStarted","Data":"081ff478b93c20e130702bfff80e555bb9c930b300a78152529606a841c89679"}
Oct 30 18:38:58.443533 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:58.442937    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49tz6" event={"ID":"bfbb88e7-35a1-4d40-9365-66a6ba00adb9","Type":"ContainerStarted","Data":"fcd918904c89c9cbb8adde1c05acef936a668104965fa77f74331af349d0c39d"}
Oct 30 18:38:58.444678 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:58.444649    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb"}
Oct 30 18:38:58.445982 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:58.445958    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64"}
Oct 30 18:38:58.562456 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:58.562410    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-49tz6" podStartSLOduration=2.7843059070000002 podStartE2EDuration="4.562392077s" totalImagesPullingTime="1.691386619s" totalInitContainerRuntime="86.699551ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:58.561822747 +0000 UTC m=+61.933397842" watchObservedRunningTime="2026-10-30 18:38:58.562392077 +0000 UTC m=+61.933967159"
Oct 30 18:38:59.284459 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.284329    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-58f458b77f-264xl"]
Oct 30 18:38:59.287832 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.287813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.344958 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.344907    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fp92n78v94iu5\""
Oct 30 18:38:59.344958 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.344928    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 18:38:59.346275 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.346249    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58f458b77f-264xl"]
Oct 30 18:38:59.370175 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-client-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.370311 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-client-certs\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.370311 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-audit-log\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.370311 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.370446 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-tls\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.370446 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-metrics-server-audit-profiles\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.370529 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.370477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-kube-api-access-7tnhb\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.442366 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.442333    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 18:38:59.442366 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.442351    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 18:38:59.442559 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.442384    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zrdzz\""
Oct 30 18:38:59.442559 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.442419    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 18:38:59.453887 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.453753    2580 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e" exitCode=0
Oct 30 18:38:59.453887 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.453791    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e"}
Oct 30 18:38:59.472030 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-kube-api-access-7tnhb\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.472147 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-client-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.472228 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-client-certs\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.472228 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-audit-log\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.472228 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.472390 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-tls\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.472390 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.472345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-metrics-server-audit-profiles\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.473276 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.473229    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-audit-log\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.473560 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.473535    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.474262 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.474240    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-metrics-server-audit-profiles\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.475780 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.475757    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-client-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.475780 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.475777    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-tls\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.476230 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.476213    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-client-certs\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.534394 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.534366    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-kube-api-access-7tnhb\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:38:59.600649 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:38:59.600576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:00.020140 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:39:00.020110    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166c00de_9f0d_49c7_8cd8_37c99b9af0a7.slice/crio-8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55 WatchSource:0}: Error finding container 8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55: Status 404 returned error can't find the container with id 8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55
Oct 30 18:39:00.021213 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.021175    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58f458b77f-264xl"]
Oct 30 18:39:00.096661 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.096641    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"]
Oct 30 18:39:00.101268 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.101252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.154037 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.153982    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 18:39:00.154204 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.154185    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 18:39:00.154317 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.154266    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kgr5v\""
Oct 30 18:39:00.154317 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.154281    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 18:39:00.154317 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.154266    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 18:39:00.154724 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.154705    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 18:39:00.176063 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.176041    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"]
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.178950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-metrics-client-ca\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76fb\" (UniqueName: \"kubernetes.io/projected/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-kube-api-access-p76fb\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.180608 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.179649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-federate-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.272168 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.272084    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 18:39:00.280461 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280620 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76fb\" (UniqueName: \"kubernetes.io/projected/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-kube-api-access-p76fb\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280620 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-federate-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280620 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280620 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280620 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-metrics-client-ca\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280878 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.280878 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.280848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.281356 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.281301    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.281630 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.281610    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.281706 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.281641    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-metrics-client-ca\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.283241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.283219    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-federate-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.283427 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.283389    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.283765 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.283747    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.283956 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.283939    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.342659 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.342633    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76fb\" (UniqueName: \"kubernetes.io/projected/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-kube-api-access-p76fb\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.422361 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.422328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:00.460418 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.460380    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"ef8aa86b3a158f1e6d435183d5937e14f1bce670cfdfd3725e963aea043c7a2a"}
Oct 30 18:39:00.460854 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.460507    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"aca63698437d29516e89da9d87d7ee0126949706a987368d346b0e454f51aa13"}
Oct 30 18:39:00.460854 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.460550    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"21df9929e1b9ab16cc41d3624db76990c08049f7f3787768ecac2dd410208eba"}
Oct 30 18:39:00.461894 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.461871    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58f458b77f-264xl" event={"ID":"166c00de-9f0d-49c7-8cd8-37c99b9af0a7","Type":"ContainerStarted","Data":"8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55"}
Oct 30 18:39:00.845925 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:00.845895    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"]
Oct 30 18:39:00.848112 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:39:00.848087    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf768f332_b6fe_48bb_8e63_ea57b00f0b7d.slice/crio-59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e WatchSource:0}: Error finding container 59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e: Status 404 returned error can't find the container with id 59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e
Oct 30 18:39:01.468160 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:01.468125    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw" event={"ID":"f768f332-b6fe-48bb-8e63-ea57b00f0b7d","Type":"ContainerStarted","Data":"59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e"}
Oct 30 18:39:02.474766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.474716    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58f458b77f-264xl" event={"ID":"166c00de-9f0d-49c7-8cd8-37c99b9af0a7","Type":"ContainerStarted","Data":"fe81792005509bfd6068c071eafe19a24296f70230b02147b1901ddf5c3a40fd"}
Oct 30 18:39:02.479014 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.478985    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e"}
Oct 30 18:39:02.479165 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.479055    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e"}
Oct 30 18:39:02.479165 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.479094    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e"}
Oct 30 18:39:02.479165 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.479138    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e"}
Oct 30 18:39:02.479316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.479173    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86"}
Oct 30 18:39:02.479316 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.479214    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerStarted","Data":"7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3"}
Oct 30 18:39:02.482891 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.482866    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"4fef51f336f3cd88f28d9b1c4ee35a19677434227d6c32de15a31af600e242a3"}
Oct 30 18:39:02.482984 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.482925    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"02f35a6219eed49f262ef515f9ece43267d4c97e777d6e4e9a94956c5d5f799c"}
Oct 30 18:39:02.482984 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.482956    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" event={"ID":"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3","Type":"ContainerStarted","Data":"e1412d4fb680f9401a90c322641bf8000a4435bfbbb8f55e21cb73eef6237df6"}
Oct 30 18:39:02.483078 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.483033    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:02.594424 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.594315    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-58f458b77f-264xl" podStartSLOduration=1.629971753 podStartE2EDuration="3.594295434s" totalImagesPullingTime="1.964323681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:02.592992437 +0000 UTC m=+65.964567518" watchObservedRunningTime="2026-10-30 18:39:02.594295434 +0000 UTC m=+65.965870518"
Oct 30 18:39:02.719375 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.719306    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z" podStartSLOduration=3.011997434 podStartE2EDuration="6.719288723s" totalImagesPullingTime="3.707291289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:02.717737362 +0000 UTC m=+66.089312443" watchObservedRunningTime="2026-10-30 18:39:02.719288723 +0000 UTC m=+66.090863804"
Oct 30 18:39:02.855070 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:02.854981    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9022269769999998 podStartE2EDuration="6.85496209s" totalImagesPullingTime="3.930112644s" totalInitContainerRuntime="22.622469ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:02.854066068 +0000 UTC m=+66.225641170" watchObservedRunningTime="2026-10-30 18:39:02.85496209 +0000 UTC m=+66.226537173"
Oct 30 18:39:03.006486 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.006446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:03.006674 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.006562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:03.058424 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.058391    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:39:03.058633 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.058618    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:39:03.069979 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.069950    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:03.070072 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.069950    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:03.107501 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.107457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:03.159211 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.159186    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:39:03.219484 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.219461    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:39:03.231105 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.231076    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:03.346010 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.345982    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swx9z\""
Oct 30 18:39:03.353299 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.353277    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l87lm\""
Oct 30 18:39:03.354347 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.354330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:03.361724 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.361681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:03.364833 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.364817    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dxlbz\""
Oct 30 18:39:03.373112 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.373090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:03.488783 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.488751    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw" event={"ID":"f768f332-b6fe-48bb-8e63-ea57b00f0b7d","Type":"ContainerStarted","Data":"f316ee6f0d2803261a98af16da23bec1ec5557b6b2c1172f29cd594c69469493"}
Oct 30 18:39:03.489138 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.488838    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw" event={"ID":"f768f332-b6fe-48bb-8e63-ea57b00f0b7d","Type":"ContainerStarted","Data":"b4fdb0d5400c102718fc3f65582caaaff8c31423e519114f4543aebaa06da264"}
Oct 30 18:39:03.489138 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.488861    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw" event={"ID":"f768f332-b6fe-48bb-8e63-ea57b00f0b7d","Type":"ContainerStarted","Data":"433be899c57e6c6df8f91dae80e651265805961133c86456858a31c561662311"}
Oct 30 18:39:03.579208 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.579177    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:03.584272 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.584248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.616188 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.615074    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw" podStartSLOduration=1.393526439 podStartE2EDuration="3.615055993s" totalImagesPullingTime="2.221529554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:03.613719488 +0000 UTC m=+66.985294570" watchObservedRunningTime="2026-10-30 18:39:03.615055993 +0000 UTC m=+66.986631077"
Oct 30 18:39:03.620474 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620619 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620619 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620619 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620738 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620775 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620775 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620855 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620912 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.620984 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621019 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.620992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621019 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621079 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621079 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621143 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621204 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621241 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621271 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621313 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.621343 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.621330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.639235 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639180    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 18:39:03.639235 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639203    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 18:39:03.639235 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639231    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 18:39:03.639483 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639461    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 18:39:03.639637 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639540    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 18:39:03.639637 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639540    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 18:39:03.639637 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639610    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 18:39:03.639814 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639676    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akuufua83pjtp\""
Oct 30 18:39:03.639888 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.639852    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 18:39:03.664033 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.664011    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 18:39:03.664785 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.664770    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 18:39:03.689742 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.689717    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 18:39:03.689948 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.689937    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gg474\""
Oct 30 18:39:03.690199 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.690187    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 18:39:03.690245 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.690220    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 18:39:03.690943 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.690921    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 18:39:03.715817 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.715793    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:03.722689 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722766 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722991 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722991 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722991 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.722991 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.722978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723178 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723178 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723178 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723178 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723178 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723178 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723469 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723469 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723469 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723469 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723469 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723469 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723375    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723804 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723697    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.723804 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.723728    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.724234 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.724156    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.724441 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.724418    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.725307 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.725080    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.726923 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.726898    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.727760 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.727733    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.727909 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.727884    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.728049 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.727889    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.728143 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.728109    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.728237 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.728188    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.728487 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.728241    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.728673 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.728652    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.728782 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.728759    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.729278 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.729256    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.729416 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.729271    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.729694 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.729678    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.730481 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.730465    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.733108 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:39:03.733087    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca86dc7_9dba_44fa_af4b_209629079b49.slice/crio-4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933 WatchSource:0}: Error finding container 4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933: Status 404 returned error can't find the container with id 4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933
Oct 30 18:39:03.733910 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.733891    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qd6bc"]
Oct 30 18:39:03.779887 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:39:03.779859    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac3d6d9_3493_46b4_a5df_e1a2b3e7585b.slice/crio-1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf WatchSource:0}: Error finding container 1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf: Status 404 returned error can't find the container with id 1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf
Oct 30 18:39:03.780857 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.780835    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77qht"]
Oct 30 18:39:03.782670 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:39:03.782649    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9fb057_17de_4446_a7cf_0730aaa6c46b.slice/crio-abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab WatchSource:0}: Error finding container abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab: Status 404 returned error can't find the container with id abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab
Oct 30 18:39:03.784255 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.784230    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdb7t"]
Oct 30 18:39:03.790237 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.790219    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:03.917633 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:03.917544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:04.350375 ip-10-0-134-133 kubenswrapper[2580]: W1030 18:39:04.350333    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa625c_734a_4009_b534_c1acc4c247ee.slice/crio-eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b WatchSource:0}: Error finding container eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b: Status 404 returned error can't find the container with id eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b
Oct 30 18:39:04.351786 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.351763    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:04.493808 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.493770    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd6bc" event={"ID":"fca86dc7-9dba-44fa-af4b-209629079b49","Type":"ContainerStarted","Data":"4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933"}
Oct 30 18:39:04.494922 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.494903    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdb7t" event={"ID":"9a9fb057-17de-4446-a7cf-0730aaa6c46b","Type":"ContainerStarted","Data":"abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab"}
Oct 30 18:39:04.496026 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.496002    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77qht" event={"ID":"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b","Type":"ContainerStarted","Data":"1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf"}
Oct 30 18:39:04.497478 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.497458    2580 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550" exitCode=0
Oct 30 18:39:04.497564 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.497545    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"}
Oct 30 18:39:04.497647 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:04.497631    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b"}
Oct 30 18:39:05.508262 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:05.508168    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd6bc" event={"ID":"fca86dc7-9dba-44fa-af4b-209629079b49","Type":"ContainerStarted","Data":"91df69991feb2f18bef703545313c15629762b59edc71913df44f80a3b40645e"}
Oct 30 18:39:06.515987 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:06.515942    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd6bc" event={"ID":"fca86dc7-9dba-44fa-af4b-209629079b49","Type":"ContainerStarted","Data":"15faa26d39b424b23164deab66aba5f05e4781032486e37d34b9f2c439450985"}
Oct 30 18:39:06.633735 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:06.633675    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qd6bc" podStartSLOduration=67.038858046 podStartE2EDuration="1m8.633658272s" totalImagesPullingTime="1.594800226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:06.633503664 +0000 UTC m=+70.005078746" watchObservedRunningTime="2026-10-30 18:39:06.633658272 +0000 UTC m=+70.005233364"
Oct 30 18:39:08.498484 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:08.498407    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:08.528097 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:08.528057    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdb7t" event={"ID":"9a9fb057-17de-4446-a7cf-0730aaa6c46b","Type":"ContainerStarted","Data":"2f29b5f1af877f3277175e8aa186dd6dfb742a343b7889f39c15f0acfbf6c3d8"}
Oct 30 18:39:08.528252 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:08.528202    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:08.844126 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:08.843798    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mdb7t" podStartSLOduration=66.412905303 podStartE2EDuration="1m10.843782951s" totalImagesPullingTime="4.430877648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:08.842389071 +0000 UTC m=+72.213964168" watchObservedRunningTime="2026-10-30 18:39:08.843782951 +0000 UTC m=+72.215358031"
Oct 30 18:39:09.536904 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.536873    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"}
Oct 30 18:39:09.537325 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.536957    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"}
Oct 30 18:39:09.537325 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.536990    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"}
Oct 30 18:39:09.537325 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.537012    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"}
Oct 30 18:39:09.537325 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.537041    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"}
Oct 30 18:39:09.537325 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.537076    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerStarted","Data":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"}
Oct 30 18:39:09.673614 ip-10-0-134-133 kubenswrapper[2580]: I1030 18:39:09.673492    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.222732787 podStartE2EDuration="6.673469572s" totalImagesPullingTime="4.435054157s" totalInitContainerRuntime="15.682628ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:09.671385867 +0000 UTC m=+73.042960950" watchObservedRunningTime="2026-10-30 18:39:09.673469572 +0000 UTC m=+73.045044654"
Oct 30 18:39:10.122451 ip-10-0-134-133 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:39:10.144234 ip-10-0-134-133 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:39:10.144409 ip-10-0-134-133 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:39:10.144472 ip-10-0-134-133 systemd[1]: kubelet.service: Consumed 4.416s CPU time.
Oct 30 18:39:10.156278 ip-10-0-134-133 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:39:10.247366 ip-10-0-134-133 kubenswrapper[5972]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:39:10.247366 ip-10-0-134-133 kubenswrapper[5972]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:39:10.247366 ip-10-0-134-133 kubenswrapper[5972]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:39:10.247366 ip-10-0-134-133 kubenswrapper[5972]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:39:10.248782 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248690    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.248782 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248782    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248787    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248791    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248794    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248797    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248800    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248803    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248806    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248808    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248811    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248814    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248817    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248820    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248828    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248831    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248834    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248836    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248839    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248842    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248845    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.248899 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248848    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248851    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248854    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248856    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248859    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248862    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248865    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248868    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248870    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248876    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248879    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248882    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248885    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248888    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248890    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248893    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248896    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248899    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248902    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248905    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.249371 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248908    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248911    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248914    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248916    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248919    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248922    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248925    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248927    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248930    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248933    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248935    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248938    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248941    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248943    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248946    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248948    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248951    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248955    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248957    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248960    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.249845 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248963    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248968    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248971    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248973    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248976    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248979    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248981    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248984    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248987    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248989    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248994    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.248998    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249001    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249004    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249007    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249009    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249012    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249015    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249018    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.250347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249021    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249024    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249026    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249029    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249031    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249034    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249037    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249040    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249044    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249047    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249050    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249053    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249056    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249059    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249063    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249066    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249068    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249071    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249074    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.250818 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249076    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249079    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249081    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249084    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249087    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249089    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249094    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249870    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249878    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249883    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249887    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249891    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249894    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249902    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249905    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249908    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249910    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249913    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249917    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.251259 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249920    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249923    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249925    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249928    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249931    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249933    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249937    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249940    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249943    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249946    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249948    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249951    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249953    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249956    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249959    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249961    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249964    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249967    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249969    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.251737 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249972    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249975    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249977    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249980    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249983    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249985    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249988    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249990    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249993    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249996    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.249998    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250001    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250004    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250006    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250009    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250012    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250014    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250017    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250019    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.252198 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250022    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250025    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250028    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250031    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250033    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250036    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250039    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250041    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250044    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250046    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250049    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250052    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250054    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250057    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250060    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250062    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250065    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250068    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250070    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250073    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.252660 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250075    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250078    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250081    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250084    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250087    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250090    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250093    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250095    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250098    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250101    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250104    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250107    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250111    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250113    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250116    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250118    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250121    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250124    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250126    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250129    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.253128 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250132    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250134    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250137    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250140    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250142    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250145    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250148    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250150    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250153    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250155    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250158    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250161    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250164    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250166    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250169    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250174    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250677    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250684    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250687    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.253614 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250690    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250693    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250696    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250698    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250701    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250704    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250708    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250712    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250715    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250719    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250722    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250725    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250727    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250730    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250733    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250735    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250738    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250744    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250747    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250749    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.254070 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250752    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250755    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250757    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250760    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250763    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250766    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250768    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250771    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250775    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250777    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250780    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250783    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250786    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250788    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250791    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250794    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250800    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250802    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250805    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250808    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.254525 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250810    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250813    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250815    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250818    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250821    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250824    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250827    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250830    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250832    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250836    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250839    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250842    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250844    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250847    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250850    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250853    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250855    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250858    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250861    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250864    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.255068 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250866    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250870    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250872    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250875    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250878    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250880    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250883    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250886    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250889    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250892    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250895    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250898    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250900    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250903    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250905    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250908    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250911    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250914    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250916    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.255634 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250919    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250921    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250927    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250929    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250932    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250936    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250940    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250943    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250946    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250948    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250951    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250954    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250957    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250960    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250962    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250974    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250977    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250980    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250982    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.256091 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250985    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250989    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250992    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250994    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.250997    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251105    5972 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251114    5972 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251120    5972 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251125    5972 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251130    5972 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251133    5972 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251138    5972 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251142    5972 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251145    5972 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251149    5972 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251155    5972 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251158    5972 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251162    5972 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251165    5972 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251168    5972 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251171    5972 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251174    5972 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251177    5972 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:39:10.256541 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251182    5972 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251185    5972 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251189    5972 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251192    5972 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251196    5972 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251200    5972 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251204    5972 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251207    5972 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251210    5972 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251214    5972 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251217    5972 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251220    5972 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251223    5972 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251226    5972 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251230    5972 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251234    5972 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251236    5972 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251239    5972 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251243    5972 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251245    5972 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251250    5972 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251253    5972 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251267    5972 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251271    5972 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:39:10.257096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251275    5972 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251280    5972 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251283    5972 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251287    5972 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251290    5972 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251293    5972 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251296    5972 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251299    5972 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251302    5972 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251305    5972 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251308    5972 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251311    5972 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251316    5972 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251319    5972 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251322    5972 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251326    5972 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251329    5972 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251332    5972 flags.go:64] FLAG: --help="false"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251335    5972 flags.go:64] FLAG: --hostname-override="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251339    5972 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251349    5972 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251354    5972 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251358    5972 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:39:10.257671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251361    5972 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251364    5972 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251367    5972 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251370    5972 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251373    5972 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251377    5972 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251380    5972 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251383    5972 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251386    5972 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251390    5972 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251394    5972 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251396    5972 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251399    5972 flags.go:64] FLAG: --lock-file=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251402    5972 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251405    5972 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251408    5972 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251414    5972 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251417    5972 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251420    5972 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251423    5972 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251426    5972 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251431    5972 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251434    5972 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251437    5972 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251442    5972 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:39:10.258196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251446    5972 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251450    5972 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251453    5972 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251456    5972 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251460    5972 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251463    5972 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251466    5972 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251469    5972 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251471    5972 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251479    5972 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251482    5972 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251485    5972 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251488    5972 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251491    5972 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251494    5972 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251497    5972 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251502    5972 flags.go:64] FLAG: --port="10250"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251505    5972 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251508    5972 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d20d98005a218369"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251512    5972 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251515    5972 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251518    5972 flags.go:64] FLAG: --register-node="true"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251521    5972 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251525    5972 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251528    5972 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251531    5972 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:39:10.258871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251534    5972 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251538    5972 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251542    5972 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251546    5972 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251549    5972 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251552    5972 flags.go:64] FLAG: --runonce="false"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251556    5972 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251562    5972 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251566    5972 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251569    5972 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251572    5972 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251575    5972 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251592    5972 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251596    5972 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251599    5972 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251602    5972 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251604    5972 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251607    5972 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251611    5972 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251614    5972 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251617    5972 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251623    5972 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251627    5972 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251630    5972 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251635    5972 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:39:10.259895 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251638    5972 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251641    5972 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251644    5972 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251647    5972 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251650    5972 flags.go:64] FLAG: --v="2"
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251655    5972 flags.go:64] FLAG: --version="false"
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251659    5972 flags.go:64] FLAG: --vmodule=""
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251663    5972 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.251666    5972 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251804    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251808    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251811    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251814    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251817    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251820    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251823    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251826    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251828    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251831    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251834    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251836    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251839    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.260857 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251842    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251845    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251847    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251850    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251852    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251855    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251858    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251863    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251866    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251869    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251871    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251874    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251876    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251879    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251882    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251884    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251887    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251890    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251892    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.261564 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251896    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251899    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251902    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251904    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251907    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251909    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251912    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251915    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251918    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251921    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251923    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251926    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251928    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251931    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251933    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251936    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251938    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251941    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251943    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251946    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.262047 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251949    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251952    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251955    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251957    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251960    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251963    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251965    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251968    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251970    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251973    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251976    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251978    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251982    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251985    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251988    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251990    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251993    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251996    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.251998    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.262529 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252002    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252007    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252010    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252014    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252016    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252019    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252022    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252024    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252027    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252030    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252032    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252035    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252037    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252041    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252044    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252047    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252049    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252052    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252055    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252057    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.263094 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252060    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252062    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252065    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252067    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252070    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252074    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252076    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252079    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252082    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252084    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252087    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252091    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252094    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252097    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252101    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.252127    5972 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:39:10.263590 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252163    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252168    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252171    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252174    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252177    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252180    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252182    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252185    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252187    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252190    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252195    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252197    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252200    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252202    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252205    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252208    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252211    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252213    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252216    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252219    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.263985 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252221    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252225    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252228    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252230    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252233    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252236    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252238    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252241    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252244    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252247    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252249    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252252    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252255    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252257    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252261    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252265    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252268    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252271    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252273    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252276    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.264646 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252278    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252281    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252285    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252287    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252290    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252292    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252295    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252298    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252300    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252303    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252306    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252308    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252311    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252316    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252318    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252321    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252323    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252326    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252328    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252331    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.265393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252333    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252336    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252339    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252341    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252346    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252350    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252353    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252356    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252360    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252363    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252365    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252368    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252371    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252373    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252377    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252380    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252383    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252386    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252388    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.266077 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252391    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252394    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252396    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252399    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252402    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252404    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252408    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252411    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252414    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252416    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252419    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252422    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252424    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252427    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252430    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252432    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252435    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252437    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252440    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.266521 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252443    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252447    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252449    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252452    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252455    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252457    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252460    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.252462    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.257065    5972 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.257082    5972 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.257099    5972 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.257107    5972 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257180    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257188    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257193    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257196    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257199    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257202    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257205    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.267058 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257209    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257213    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257216    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257219    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257222    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257224    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257227    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257231    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257234    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257237    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257239    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257242    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257256    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257261    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257265    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257270    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257274    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257277    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257280    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.267555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257283    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257285    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257288    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257291    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257293    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257296    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257299    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257301    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257304    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257307    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257310    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257312    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257316    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257318    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257321    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257324    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257326    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257330    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257332    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.268113 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257336    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257340    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257345    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257349    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257354    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257357    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257360    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257368    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257371    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257374    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257376    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257379    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257381    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257384    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257387    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257389    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257392    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257394    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257397    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257400    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.268620 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257402    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257405    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257407    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257410    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257412    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257415    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257419    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257424    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257428    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257432    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257437    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257440    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257443    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257445    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257448    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257451    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257453    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257456    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257459    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257461    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.269085 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257469    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257472    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257475    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257479    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257484    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257487    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257490    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257493    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257495    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257498    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257502    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257506    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257510    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257515    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257520    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257522    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257525    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257528    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257531    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.269555 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257533    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257536    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.257564    5972 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257781    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257788    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257791    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257794    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257797    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257800    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257803    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257806    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257810    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257813    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257816    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257825    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257828    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.270084 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257832    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257836    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257841    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257845    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257849    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257853    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257855    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257858    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257860    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257863    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257866    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257869    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257871    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257874    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257876    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257887    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257889    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257892    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257895    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.270460 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257898    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257900    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257903    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257905    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257908    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257911    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257914    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257918    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257923    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257928    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257932    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257936    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257944    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257947    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257950    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257953    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257955    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257958    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257960    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257963    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.271204 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257966    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257969    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257971    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257973    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257976    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257979    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257982    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257985    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257987    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257990    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257992    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.257995    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258001    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258005    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258010    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258014    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258018    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258021    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258024    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.271868 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258028    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258032    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258035    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258038    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258041    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258044    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258052    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258055    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258058    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258060    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258063    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258066    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258068    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258071    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258073    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258076    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258080    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258085    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258089    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258094    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.272330 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258098    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258101    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258104    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258106    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258109    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258111    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258114    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258116    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258119    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258122    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258124    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258126    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258129    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258132    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258134    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.258159    5972 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:39:10.272871 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258208    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258212    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258215    5972 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258224    5972 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258227    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258230    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258232    5972 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258235    5972 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258237    5972 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258240    5972 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258244    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258248    5972 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258253    5972 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258257    5972 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258262    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258269    5972 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258272    5972 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258274    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258277    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258279    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:39:10.273249 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258282    5972 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258285    5972 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258287    5972 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258290    5972 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258292    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258294    5972 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258297    5972 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258299    5972 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258302    5972 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258305    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258307    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258310    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258312    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258315    5972 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258318    5972 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258322    5972 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258332    5972 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258338    5972 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258343    5972 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:39:10.273739 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258347    5972 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258350    5972 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258353    5972 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258355    5972 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258358    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258361    5972 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258363    5972 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258366    5972 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258370    5972 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258373    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258376    5972 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258378    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258381    5972 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258383    5972 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258386    5972 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258388    5972 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258391    5972 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258393    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258396    5972 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:39:10.274266 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258399    5972 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258401    5972 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258404    5972 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258406    5972 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258410    5972 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258415    5972 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258420    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258427    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258430    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258433    5972 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258435    5972 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258444    5972 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258447    5972 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258449    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258452    5972 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258455    5972 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258458    5972 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258460    5972 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258463    5972 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258465    5972 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:39:10.274783 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258468    5972 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258473    5972 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258476    5972 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258478    5972 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258481    5972 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258484    5972 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258486    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258495    5972 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258500    5972 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258504    5972 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258509    5972 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258513    5972 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258515    5972 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258518    5972 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258520    5972 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258523    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258526    5972 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258528    5972 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258531    5972 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:39:10.275393 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258535    5972 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258540    5972 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258543    5972 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258547    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258550    5972 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258559    5972 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258564    5972 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258567    5972 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:10.258570    5972 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.258652    5972 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.261619    5972 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.261700    5972 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.261971    5972 server.go:1014] "Starting client certificate rotation"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.262108    5972 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.262158    5972 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:57 +0000 UTC" deadline="2028-06-18 21:49:11.408900097 +0000 UTC"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.262197    5972 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14331h10m1.146705216s"
Oct 30 18:39:10.276003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.262458    5972 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:39:10.276548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.263751    5972 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:39:10.276548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.265274    5972 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:39:10.276548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.267285    5972 log.go:25] "Validated CRI v1 image API"
Oct 30 18:39:10.276548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.268020    5972 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:39:10.276548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.270375    5972 fs.go:135] Filesystem UUIDs: map[541265af-cc59-44b0-a9ad-2826cd2f94a9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b7c8123e-01b2-4a3e-9a6b-85d85a09d367:/dev/nvme0n1p3]
Oct 30 18:39:10.276730 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.270396    5972 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf/userdata/shm major:0 minor:462 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933/userdata/shm major:0 minor:456 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55/userdata/shm major:0 minor:360 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b/userdata/shm major:0 minor:484 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/alertmanager/9 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/config-reloader/6 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~empty-dir/config-out major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~projected/kube-api-access-dmc9l:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~projected/kube-api-access-dmc9l major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~projected/tls-assets major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/config-volume major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/web-config major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volumes/kubernetes.io~projected/kube-api-access-xhtft:{mountpoint:/var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volumes/kubernetes.io~projected/kube-api-access-xhtft major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~projected/kube-api-access-7tnhb:{mountpoint:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~projected/kube-api-access-7tnhb major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47825bca-e2dc-480b-b29e-3e071b260fd9/volumes/kubernetes.io~projected/kube-api-access-5dwsx:{mountpoint:/var/lib/kubelet/pods/47825bca-e2dc-480b-b29e-3e071b260fd9/volumes/kubernetes.io~projected/kube-api-access-5dwsx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47825bca-e2dc-480b-b29e-3e071b260fd9/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/47825bca-e2dc-480b-b29e-3e071b260fd9/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~projected/kube-api-access-2tt5k:{mountpoint:/var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~projected/kube-api-access-2tt5k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~projected/kube-api-access-b9f7t:{mountpoint:/var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~projected/kube-api-access-b9f7t major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~projected/kube-api-access-vwrcr:{mountpoint:/var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~projected/kube-api-access-vwrcr major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57299938-b4d9-45b6-9650-e0f0c9b50c80/volumes/kubernetes.io~projected/kube-api-access-7rkhk:{mountpoint:/var/lib/kubelet/pods/57299938-b4d9-45b6-9650-e0f0c9b50c80/volumes/kubernetes.io~projected/kube-api-access-7rkhk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57299938-b4d9-45b6-9650-e0f0c9b50c80/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/57299938-b4d9-45b6-9650-e0f0c9b50c80/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f1b72b8-3c48-4426-b83c-94bc84b69bf4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5f1b72b8-3c48-4426-b83c-94bc84b69bf4/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~projected/kube-api-access-xwrhq:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~projected/kube-api-access-xwrhq major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/config-reloader/2 major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/prometheus/15 major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~empty-dir/config-out major:0 minor:475 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~projected/kube-api-access-76p4w:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~projected/kube-api-access-76p4w major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~projected/tls-assets major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/config major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:476 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/web-config major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a9fb057-17de-4446-a7cf-0730aaa6c46b/volumes/kubernetes.io~projected/kube-api-access-jn4cw:{mountpoint:/var/lib/kubelet/pods/9a9fb057-17de-4446-a7cf-0730aaa6c46b/volumes/kubernetes.io~projected/kube-api-access-jn4cw major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d87c914-b46f-4c4e-a7d9-3eb4335c463d/volumes/kubernetes.io~projected/kube-api-access-crbn8:{mountpoint:/var/lib/kubelet/pods/9d87c914-b46f-4c4e-a7d9-3eb4335c463d/volumes/kubernetes.io~projected/kube-api-access-crbn8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8924fe8-5503-4a1d-a361-18221c829534/volumes/kubernetes.io~projected/kube-api-access-cb8ch:{mountpoint:/var/lib/kubelet/pods/b8924fe8-5503-4a1d-a361-18221c829534/volumes/kubernetes.io~projected/kube-api-access-cb8ch major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~projected/kube-api-access-69nwv:{mountpoint:/var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~projected/kube-api-access-69nwv major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc0a19a4-6a53-4252-8deb-30327d884859/volumes/kubernetes.io~projected/kube-api-access-pm4gz:{mountpoint:/var/lib/kubelet/pods/cc0a19a4-6a53-4252-8deb-30327d884859/volumes/kubernetes.io~projected/kube-api-access-pm4gz major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc0a19a4-6a53-4252-8deb-30327d884859/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cc0a19a4-6a53-4252-8deb-30327d884859/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd48639e-7f2c-4383-9716-29c1b4e68994/volumes/kubernetes.io~projected/kube-api-access-pm4sb:{mountpoint:/var/lib/kubelet/pods/cd48639e-7f2c-4383-9716-29c1b4e68994/volumes/kubernetes.io~projected/kube-api-access-pm4sb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~projected/kube-api-access-5kk45:{mountpoint:/var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~projected/kube-api-access-5kk45 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfd92d06-4598-48c4-822b-d0dff1df2936/volumes/kubernetes.io~projected/kube-api-access-6crfc:{mountpoint:/var/lib/kubelet/pods/cfd92d06-4598-48c4-822b-d0dff1df2936/volumes/kubernetes.io~projected/kube-api-access-6crfc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e510aafb-e0d8-4898-a6ac-6aaaac776987/volumes/kubernetes.io~projected/kube-api-access-hfml5:{mountpoint:/var/lib/kubelet/pods/e510aafb-e0d8-4898-a6ac-6aaaac776987/volumes/kubernetes.io~projected/kube-api-access-hfml5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1fea5aa-6e82-4768-8f35-fb947216fb9f/volumes/kubernetes.io~projected/kube-api-access-j7brm:{mountpoint:/var/lib/kubelet/pods/f1fea5aa-6e82-4768-8f35-fb947216fb9f/volumes/kubernetes.io~projected/kube-api-access-j7brm major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~projected/kube-api-access-p76fb:{mountpoint:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~projected/kube-api-access-p76fb major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fca86dc7-9dba-44fa-af4b-209629079b49/volumes/kubernetes.io~projected/kube-api-access-26xtl:{mountpoint:/var/lib/kubelet/pods/fca86dc7-9dba-44fa-af4b-209629079b49/volumes/kubernetes.io~projected/kube-api-access-26xtl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fca86dc7-9dba-44fa-af4b-209629079b49/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/fca86dc7-9dba-44fa-af4b-209629079b49/volumes/kubernetes.io~secret/metrics-certs major:0 minor:442 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9c876216d817ad01425adb228def303bfecedc012fec53ce84f27adbe08bba67/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f5f78d4a795567477793e1082d9e18bc133d481472cac3369e5e08bf77ba5816/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e8f85eb863f44e9d6482379bd521fbb7c9d42bc81ecdbe660861833af01217a9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9ce73a8ce5c4f4f1df6f7269d9a071796393d3509b14620522b3df6b5e3a80f3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/24f58a0340dd1fd507de1caa6c002d7d782855dfaf424ac783c57a4e46040da7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/88e3cc4478f00045b17d27334403229b116f8d3a2a39ed14f3d9183f7c4a2cf9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e990c6c19db995a6a4dc9d8f526290b185304126b2e1e401a384bb62fc154a36/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/75a795e3d8b1eb4fdad7ed54b523b997a62d54cd9da714221c0f56cc061e9c92/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/54591dfa39244a37a3b4cb68b6ed6847b3cb580c4c64dca711c6dd5a3e0221a5/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/8f727a946add44df4dcfe9088804a2959c99cd83895aab61a3a4b4f019793b85/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/7f2254fc3852150f13d0cfe90d44de9a8293f5ea6a0c611b7952d1d19051ad1e/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/245dcff091f6c62047741af4067bdb9505fd632b5be24c4c394e362997c2960e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/3a1c65769a1dc1eece0176e96d920e6d4cbd15c5d0bbfc40fc69e6a5379fb1b9/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/116cd6e4b478402c94d3cfc2403fb2860ea89616374bf2d32848df754e6bae42/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/60c646d732ded20a7e94980df84e4f1a843e2b5921936a396e47126664307291/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0afe8f15bbb67a6690d7fc5edbd66810f751b54eaeddd87d74b37dd3410129c4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ea2a282e036de0ac272e7b7de69aa85663ea1c3590144b46fde68a438a0a74be/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4756bc1daf06942209a6936c06b32e0936bf8786ea9264303116a0eddf7b0a0a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1eeca8aaf81c74a47ad090d2c88dea6be19745adb9902ace032fa30c29bd953b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2298b31ec881f1f90eb2aab94c8ae5aaa3728b20c2a432177ab56f1cb4a3a9c7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/be1976ce4a65f7479d044c5b1c2595d38bdddad53960f2c1089d32e27cfb6510/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/96eb241c05c01914cfb115fb6f049bccae0476b95425a9425c5fb47d86c16df7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/83930159ccf3019e657e4ea7f4918a295f14aa4413e2e3246b70755c81659ddd/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/90e7190be93a2be0b724def86f5bceddec2de86082d66d3cef6dc2ed5aeba44d/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/abba9f8a0e1493659d27fa6a6d6296e735816b2ae50f29599d6ec0c94e557b16/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/135054f938010f675523b2748d45c00cd3b808a4f5c58081a752f0fe7f6a97b4/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/efb4e1f1fefc90f93c5b1fe77f9400a9ba5747acdce90e028c1a90ed9b42e0a4/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/5dcb12f6891e7322b24a9ea70f2447b54861f9d4313f066af729469aa8176337/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/a757a7a04355524b9e38137661f37585e75ab92f37dd0d472fe5cdd6d7e954db/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/20bb291a2e42c3a4bed91d3a0944cba4284598494f73c5b18848a13a594d3f39/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/04260dc14f03491683f2ef99b073fe0fb577889d5ef239194130cb5b299fb1cf/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/15650136a8246d555e73268db311dd09d7b99357cac08fd81170db62d3d6526a/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/5f86d5835ca696ffc6ced68ca01a0b71cacb2cfcf0d1b8cc6206e2be3a4b9cbb/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/0556807023d230ea0b408f56702213d6243bd70a2d1cf4bb053fbd87e86aeef8/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/4fb1fabc144c551386befe8c2dea594ad9adb5a85cc12b073990b4982a3cdd02/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/aaf4c6514a8658ca417ce1aba1d693c80dd38efd4ed5e0eeda820627796cdd12/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/3a1d02c7c6c253d6fb2cbf13fe87b266cd6af299a6e7fe1f8e5d2f9115dd03f5/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a72be2f3f8447597066b1fe7cbb37a3cf9433329e2bb2d114d8c8bc41977abb4/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/7ce19a400770c6ea8736ac9a5c42162b75bdd15db36fc2b3a897a6a9e657c3e2/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/2c7d6b4126a13e964fe82ffb1c8c779d0f3c8eb4873201a3412249d004be97a1/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/0bfe678ce6302b39ab3c02f9a368a136e83f3a8827a15d6fd9e97ac02069ecce/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/af9b39944af0dd54e66a68f208b42112f5bbeb62a3a4a6b9b03ad3754d4704c7/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/c1e31c1528935da06f5f0879852cf2bea228c959361fc77f53416857ae7787c8/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/d8c6e60f9d39680dc79d6003f557e8c4f8bc140a4b117e7546db85fd9a40f250/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/d83a953f135237e059d9f77a6543560020ef26358545f5bbec5f4f79cf9805e8/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/31bd4e4cbb98dfa27de10ea64c696bb8f5fb885b2b4105d353a710a9a5479874/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/358907091a7f6d4abfdd9d3e1ff04ca263aaef35bb59e4fb66639bec18408ab9/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/f950e58b322c62d26cbe8dfd224b02859bf254fea8da93a052488a44552ae548/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/d84ec4f34b6c9f17a397da5b56365138247add54174086c4f70c376f9d2635d7/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/09a00d0df1998864f15f595ff07ecc551f7dae0d61025e269fa51127dc3b8231/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/df2f2d602160305c5a7ba96c16a8722070c00707f11ca4b1b273bd60cbd9408e/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/451b4f4d1208adcc2dd03dc1a13ffae5f635b18d56591a60dc281e29f531b9f1/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/4e704816b2338fe41e7c12b52a6f56957a9c10951113e3c4adf5db56931fab29/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/588a4aaa38ae1f7da0810cff4062bfdaa833001f91e3f639d49156284f8524e2/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/8d5628180004ad6417994f801880d171d1e0ed871e4c8a5e2ebcc686c0495ca9/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/0cd85f5197142fecd4964525b54fd27249d5fb2f2dee626ebf986b7892fcdae0/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/80373dfc596982757bdb0a12ed05e33726c77c3c6ffe8f91919260d44b88e2b4/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/5fbf2b0f8992d5aebfcc4209131aa517b6883475b8bcbf9b4b90ccf995b7145b/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/8ec81a6a196a90fc15aedab5b62ccbaa75fd4723f02a1b46632993f572dbd10c/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/59fd5beb1368aa5b9f45435be4efa0c3e46a55e8bb96fd6b5d91e7ce1324a944/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/c818dab7fabc0be61f6b813b55fc6ee61d725716a7dc643df56c74760dab3b76/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/e28a161d48532b0f08b8751f736cc4225838bbc46c83d159f9ded04a4506ca81/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/7dff79b7fcbc37d79053e32f1c4d9115671843a22396946f5fd0f3e4f9c2c89b/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/78929046af7ba46f34e46962451444143e116b4b9a117ca327f6b622db3ef795/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/b14950cbd1fec1cec09d0435d65e0d97e0e456feba51452ed041dd76890a2f4e/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/b39709472a66235b42eb6750cb751f6057c738f8331ed2ea0f958b03079347ab/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/05b3d2cb5303cea2b2ba1db993bf6d9a2ac9e423a7400e434bf2cf67b42d8d7a/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/18bfef537b3e3cbfeaa386f6c4834a6f29186f3b801699d90be0d2617648db43/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/ec70c44c0685760211d991f380ba0ccfc3732fe9acf62c2962a012192df37385/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/1abb85b7142414b8148b4df9a0e39e284e4e24d778827fe46c66a52b0037db67/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/8cbf02dc734136549f97c5f797a2c7ca2b1944826e068ecb8a99d42158d2b025/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/4dd9a346aa58da96260052e566c9da6e1874d8bb1327d22c2872250d750caa24/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/4a48028750a8fe91df0ce7de0be13489f1acbafa77e480ea7e7e1de6c3561aa0/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7c43bd69c3e1efbb179c0e39cca5e69c8b181dcbb39ea60a5b45de71f1763293/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/0a273924d55d613869b76a2cb168c94be93fbe1998d1326d8c96fedeacaaf069/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/382dfd9d84de9ed7ea0d73a06d0f0da7b7136ea8d40868ecc81fe55041620a36/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/f6ede3c01dfef3db4119ff0aa9dc05a7223dee94bc77144abd20aa8653f521c4/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/bb3a33f6f4b23bb54db5de5b7b923ddcc29bdf5034b4cc562226b3d70074de80/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b61ee36f4786fb25d1239f7e0281f87846252f2c2e88096f5f3033448b867f2d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-533:{mountpoint:/var/lib/containers/storage/overlay/3b3b2d66cb42b6d389cd83317ba691a49645ea7c3b169fd77bd049183377fbaf/merged major:0 minor:533 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a9eae2c8771d06d2936e6d0cd99627e506b71058fccf72cd90712da9c242da8f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9a7738caceba8ff0151ee1af28492230dfb265775e081b5d7201ce75324d285c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/047f14be0dc804c55e8c407a621b2a1aa6b8da9a7b09a86a435bc2900253ea86/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/9240b816717e6df807e582572132b34e16f287bdbce27d10c0da892f1499a1fc/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/55c349fcc94e22fc64a6efdf0919a72417b95c9945d0d37e0b4fcabba69449b1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/63f831289302fc72b3051b8e7405efb3cfe79d715f39904a13ae94db0defaf31/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3ca6a6bc9d5988a9897ec1d2e17674c5ba4a53f5e2e985f42b72f106928cf1fa/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fa16273493b6b5c0825b2008ffd1f6b5acd02add39e372760d4f11be7c4834e7/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:39:10.281870 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.281313    5972 manager.go:217] Machine: {Timestamp:2026-10-30 18:39:10.279971295 +0000 UTC m=+0.071938578 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101797 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec225ff72f206814640b025affea46b2 SystemUUID:ec225ff7-2f20-6814-640b-025affea46b2 BootID:de56a7de-4a2c-4141-9ade-9254a26a2001 Filesystems:[{Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb61957fc3830c3086b17ea3e9301c3d975bcf9b945488772c1bcf91456b25e5/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:306 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:471 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volumes/kubernetes.io~projected/kube-api-access-xhtft DeviceMajor:0 DeviceMinor:64 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d321f5bdc09e0b880baa8adbb26e83a633f71671afc103a6039e27b9441b7beb/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:301 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:305 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~projected/kube-api-access-5kk45 DeviceMajor:0 DeviceMinor:270 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:356 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a9fb057-17de-4446-a7cf-0730aaa6c46b/volumes/kubernetes.io~projected/kube-api-access-jn4cw DeviceMajor:0 DeviceMinor:457 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:300 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfd92d06-4598-48c4-822b-d0dff1df2936/volumes/kubernetes.io~projected/kube-api-access-6crfc DeviceMajor:0 DeviceMinor:62 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b8924fe8-5503-4a1d-a361-18221c829534/volumes/kubernetes.io~projected/kube-api-access-cb8ch DeviceMajor:0 DeviceMinor:63 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b91f598d9cf38f2daa8cb876b20c2831f936c2bce7fc2791ac98e60850e5845/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:274 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:312 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:471 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b91cb31fbb381638cf690f6cb32a20444e28b693a2548f75495a29cc04c7937d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:321 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:307 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:319 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:305 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:357 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:466 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~projected/kube-api-access-b9f7t DeviceMajor:0 DeviceMinor:252 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b/userdata/shm DeviceMajor:0 DeviceMinor:484 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cd48639e-7f2c-4383-9716-29c1b4e68994/volumes/kubernetes.io~projected/kube-api-access-pm4sb DeviceMajor:0 DeviceMinor:57 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47825bca-e2dc-480b-b29e-3e071b260fd9/volumes/kubernetes.io~projected/kube-api-access-5dwsx DeviceMajor:0 DeviceMinor:202 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0481d59f616a5ee705a165c18237ab07c9bed4af650479a16b27b27febc3a64/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:308 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:469 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:471 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57299938-b4d9-45b6-9650-e0f0c9b50c80/volumes/kubernetes.io~projected/kube-api-access-7rkhk DeviceMajor:0 DeviceMinor:204 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:304 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:475 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fca86dc7-9dba-44fa-af4b-209629079b49/volumes/kubernetes.io~projected/kube-api-access-26xtl DeviceMajor:0 DeviceMinor:67 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e510aafb-e0d8-4898-a6ac-6aaaac776987/volumes/kubernetes.io~projected/kube-api-access-hfml5 DeviceMajor:0 DeviceMinor:56 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:471 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ed2fd6b3357724013163c0be11b913c10703459aeb3a411d3d6c6ab0e772054/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:247 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:322 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/522412d0-7818-4a3d-975f-c816eb5d7a13/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:305 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f4e837d17a216ddce855bc38a1157a3497f6c666975fe09417f3dccf78baa55/userdata/shm DeviceMajor:0 DeviceMinor:360 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:438 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:313 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87a57a47d6216e4e036d0ba71ccf6780cda123e32b14c7ae8cae2e2f0c8ff90d/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59d79c10d320a5322d6719e547ae5106773f49ed353d3aeeb403e640ead3559e/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d87c914-b46f-4c4e-a7d9-3eb4335c463d/volumes/kubernetes.io~projected/kube-api-access-crbn8 DeviceMajor:0 DeviceMinor:66 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~projected/kube-api-access-69nwv DeviceMajor:0 DeviceMinor:275 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/125cc0b7251670ad588d55187fbb0d2895ca98ba64c223eaac452835347404b6/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:323 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:381 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fca86dc7-9dba-44fa-af4b-209629079b49/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:442 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a87f701cab0a1f58cf64629bbcfc7df266fd1233298d1a50bf01a9ff1bbe933/userdata/shm DeviceMajor:0 DeviceMinor:456 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:468 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9168e270b433f29d6b36c93169e05527fb37f926fa76b34eb97a0dad8eb291a1/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:383 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1fea5aa-6e82-4768-8f35-fb947216fb9f/volumes/kubernetes.io~projected/kube-api-access-j7brm DeviceMajor:0 DeviceMinor:55 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f50b52778fcf49fe1b4fc40e54b998cab24f4f362fb71262d0a46b4ae8a171b/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~projected/kube-api-access-dmc9l DeviceMajor:0 DeviceMinor:314 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2eeb57fba3e4965e96b13a56a478aef1ede6124b7414b2fef31abc635282e088/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:358 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/47825bca-e2dc-480b-b29e-3e071b260fd9/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cc0a19a4-6a53-4252-8deb-30327d884859/volumes/kubernetes.io~projected/kube-api-access-pm4gz DeviceMajor:0 DeviceMinor:203 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfbb88e7-35a1-4d40-9365-66a6ba00adb9/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:273 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~projected/kube-api-access-xwrhq DeviceMajor:0 DeviceMinor:331 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:305 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1df0d8b3b3e4d2f50be221ba21b9fdaf4aaa7a3908a340707db7843b4beea8bf/userdata/shm DeviceMajor:0 DeviceMinor:462 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:476 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf611fab-4203-425e-8db6-6ab6072ec4c2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/549292be-cfde-49a0-bd85-b1854a005bff/volumes/kubernetes.io~projected/kube-api-access-vwrcr DeviceMajor:0 DeviceMinor:278 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/166c00de-9f0d-49c7-8cd8-37c99b9af0a7/volumes/kubernetes.io~projected/kube-api-access-7tnhb DeviceMajor:0 DeviceMinor:359 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:377 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cc0a19a4-6a53-4252-8deb-30327d884859/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4619a6bc25e54bec5c4cfe3f5986c2ad9afd15e5afd7589ebf7177a3db91f82/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~projected/kube-api-access-p76fb DeviceMajor:0 DeviceMinor:384 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abe48ce6d364e2f106786b5f5f122fb7e1055316d4172121be4e95249d51b8ab/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:470 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:474 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:473 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f1b72b8-3c48-4426-b83c-94bc84b69bf4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ced792174b60859181573f2503466afc0fcd9505abdb105f3b54747315319c9/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:324 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:308 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~projected/kube-api-access-76p4w DeviceMajor:0 DeviceMinor:483 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/48d98419-9d48-4a09-96cd-940bee92d0b3/volumes/kubernetes.io~projected/kube-api-access-2tt5k DeviceMajor:0 DeviceMinor:65 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6359165a27869f16df350a18c4989849373a718cf56223da92d61671276eba0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc7f92bd418a8183fdaf5230099d32b8123c8cedf02c1d54f324d71b5c36f27f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:320 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:472 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/0808db1b-d15b-47c9-aa62-8a1a6309398b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/57299938-b4d9-45b6-9650-e0f0c9b50c80/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/f768f332-b6fe-48bb-8e63-ea57b00f0b7d/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:382 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb975e74f5c40604fe44d949a37a8d4945836332da8ccc5bfbeba50d059dc9a0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:467 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-533 DeviceMajor:0 DeviceMinor:533 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b91f598d9cf38f MacAddress:82:5a:82:6f:8c:18 Speed:10000 Mtu:8901} {Name:125cc0b7251670a MacAddress:52:a4:47:a7:8b:d7 Speed:10000 Mtu:8901} {Name:1df0d8b3b3e4d2f MacAddress:9e:6d:2f:ff:c2:05 Speed:10000 Mtu:8901} {Name:4a87f701cab0a1f MacAddress:1a:25:f6:30:12:17 Speed:10000 Mtu:8901} {Name:56f9e82c7cd9128 MacAddress:ca:74:8e:c1:b0:c3 Speed:10000 Mtu:8901} {Name:59d79c10d320a53 MacAddress:56:9f:92:42:d9:1e Speed:10000 Mtu:8901} {Name:5ced792174b6085 MacAddress:72:71:72:eb:dd:69 Speed:10000 Mtu:8901} {Name:87a57a47d6216e4 MacAddress:d2:14:e3:bc:5c:2c Speed:10000 Mtu:8901} {Name:8f4e837d17a216d MacAddress:fe:30:3c:1b:2b:64 Speed:10000 Mtu:8901} {Name:abe48ce6d364e2f MacAddress:d2:d8:70:b7:2f:37 Speed:10000 Mtu:8901} {Name:b0481d59f616a5e MacAddress:5a:7d:2e:7a:04:0b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d9:ca:9a:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:52:c2:71:ab:0e:92 Speed:0 Mtu:8901} {Name:c4619a6bc25e54b MacAddress:ba:76:d4:2c:80:a1 Speed:10000 Mtu:8901} {Name:d321f5bdc09e0b8 MacAddress:ea:48:c2:42:90:ff Speed:10000 Mtu:8901} {Name:eea95d358c5cbe2 MacAddress:e2:90:b0:80:56:71 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d9:ca:9a:cd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:95:6b:2f:81:d6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d2:92:6c:ff:30:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:39:10.281870 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.281873    5972 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:39:10.282274 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.281927    5972 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:39:10.282308 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282278    5972 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:39:10.282464 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282301    5972 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-133.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:39:10.282509 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282467    5972 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:39:10.282509 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282475    5972 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:39:10.282509 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282485    5972 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:39:10.282602 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282509    5972 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:39:10.282644 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282635    5972 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:39:10.282849 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282840    5972 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:39:10.282975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.282966    5972 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:39:10.283017 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.283008    5972 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:39:10.283045 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.283025    5972 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:39:10.283045 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.283036    5972 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:39:10.283110 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.283046    5972 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:39:10.283110 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.283055    5972 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:39:10.284337 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.284322    5972 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:39:10.284379 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.284339    5972 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:39:10.284994 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.284970    5972 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:39:10.287879 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.287865    5972 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:39:10.287879 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.287878    5972 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:39:10.287997 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.287987    5972 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:39:10.288193 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288182    5972 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:39:10.288228 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288212    5972 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:39:10.288360 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288348    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:39:10.288394 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288376    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:39:10.288394 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288386    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288394    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288401    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288408    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288413    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288420    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288427    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288432    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:39:10.288444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288440    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:39:10.288683 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288451    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:39:10.288683 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288481    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:39:10.288683 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.288490    5972 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:39:10.292053 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.292036    5972 server.go:1263] "Started kubelet"
Oct 30 18:39:10.292143 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.292092    5972 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:39:10.292383 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.292312    5972 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:39:10.292475 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.292417    5972 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:39:10.292673 ip-10-0-134-133 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:39:10.293464 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.293443    5972 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:39:10.293929 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.293850    5972 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:39:10.299730 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.299707    5972 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:39:10.299921 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.299799    5972 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:39:10.299921 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.299812    5972 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:39:10.299921 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.299832    5972 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:39:10.299921 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.299918    5972 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:39:10.300217 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.299975    5972 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:39:10.300217 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.300014    5972 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:59 +0000 UTC" deadline="2028-04-08 19:43:32.344300543 +0000 UTC"
Oct 30 18:39:10.300217 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.300033    5972 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12625h4m22.044270721s"
Oct 30 18:39:10.303047 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303029    5972 factory.go:55] Registering systemd factory
Oct 30 18:39:10.303047 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303048    5972 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303018    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303107    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303128    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303141    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303151    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303163    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549292be-cfde-49a0-bd85-b1854a005bff" volumeName="kubernetes.io/projected/549292be-cfde-49a0-bd85-b1854a005bff-kube-api-access-vwrcr" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303175    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57299938-b4d9-45b6-9650-e0f0c9b50c80" volumeName="kubernetes.io/configmap/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303189    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" volumeName="kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 18:39:10.303201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303203    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303212    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47825bca-e2dc-480b-b29e-3e071b260fd9" volumeName="kubernetes.io/configmap/47825bca-e2dc-480b-b29e-3e071b260fd9-config-volume" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303220    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303231    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303243    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48d98419-9d48-4a09-96cd-940bee92d0b3" volumeName="kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-tuned" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303252    5972 factory.go:153] Registering CRI-O factory
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303268    5972 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303312    5972 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303329    5972 factory.go:103] Registering Raw factory
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303351    5972 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303256    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303573    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303618    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/projected/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-kube-api-access-7tnhb" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303639    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 18:39:10.303656 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303660    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" volumeName="kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303678    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd48639e-7f2c-4383-9716-29c1b4e68994" volumeName="kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303696    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfd92d06-4598-48c4-822b-d0dff1df2936" volumeName="kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303712    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549292be-cfde-49a0-bd85-b1854a005bff" volumeName="kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303727    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57299938-b4d9-45b6-9650-e0f0c9b50c80" volumeName="kubernetes.io/empty-dir/57299938-b4d9-45b6-9650-e0f0c9b50c80-data-volume" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303740    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303752    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="522412d0-7818-4a3d-975f-c816eb5d7a13" volumeName="kubernetes.io/configmap/522412d0-7818-4a3d-975f-c816eb5d7a13-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303765    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0808db1b-d15b-47c9-aa62-8a1a6309398b" volumeName="kubernetes.io/projected/0808db1b-d15b-47c9-aa62-8a1a6309398b-kube-api-access-xhtft" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303780    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/empty-dir/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-tmp" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303795    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303809    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303828    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47825bca-e2dc-480b-b29e-3e071b260fd9" volumeName="kubernetes.io/empty-dir/47825bca-e2dc-480b-b29e-3e071b260fd9-tmp-dir" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303855    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b" volumeName="kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303888    5972 manager.go:319] Starting recovery of all containers
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303893    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87c914-b46f-4c4e-a7d9-3eb4335c463d" volumeName="kubernetes.io/projected/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kube-api-access-crbn8" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303910    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303924    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549292be-cfde-49a0-bd85-b1854a005bff" volumeName="kubernetes.io/empty-dir/549292be-cfde-49a0-bd85-b1854a005bff-volume-directive-shadow" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303936    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf611fab-4203-425e-8db6-6ab6072ec4c2" volumeName="kubernetes.io/configmap/cf611fab-4203-425e-8db6-6ab6072ec4c2-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303951    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303964    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48d98419-9d48-4a09-96cd-940bee92d0b3" volumeName="kubernetes.io/projected/48d98419-9d48-4a09-96cd-940bee92d0b3-kube-api-access-2tt5k" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.303990    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57299938-b4d9-45b6-9650-e0f0c9b50c80" volumeName="kubernetes.io/secret/57299938-b4d9-45b6-9650-e0f0c9b50c80-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304004    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-grpc-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304018    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304032    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1fea5aa-6e82-4768-8f35-fb947216fb9f" volumeName="kubernetes.io/empty-dir/f1fea5aa-6e82-4768-8f35-fb947216fb9f-tmp-dir" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304043    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fca86dc7-9dba-44fa-af4b-209629079b49" volumeName="kubernetes.io/projected/fca86dc7-9dba-44fa-af4b-209629079b49-kube-api-access-26xtl" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304056    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549292be-cfde-49a0-bd85-b1854a005bff" volumeName="kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304068    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304083    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-client-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304098    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304112    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304125    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304140    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-federate-client-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304153    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="522412d0-7818-4a3d-975f-c816eb5d7a13" volumeName="kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304165    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304177    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="522412d0-7818-4a3d-975f-c816eb5d7a13" volumeName="kubernetes.io/projected/522412d0-7818-4a3d-975f-c816eb5d7a13-kube-api-access-b9f7t" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304192    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549292be-cfde-49a0-bd85-b1854a005bff" volumeName="kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304204    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304216    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1fea5aa-6e82-4768-8f35-fb947216fb9f" volumeName="kubernetes.io/projected/f1fea5aa-6e82-4768-8f35-fb947216fb9f-kube-api-access-j7brm" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304228    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0808db1b-d15b-47c9-aa62-8a1a6309398b" volumeName="kubernetes.io/secret/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304277    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/empty-dir/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-audit-log" seLinuxMountContext=""
Oct 30 18:39:10.304282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304302    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc0a19a4-6a53-4252-8deb-30327d884859" volumeName="kubernetes.io/projected/cc0a19a4-6a53-4252-8deb-30327d884859-kube-api-access-pm4gz" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304315    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304328    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-client-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304342    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/projected/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-kube-api-access-xwrhq" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304353    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304364    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304377    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e510aafb-e0d8-4898-a6ac-6aaaac776987" volumeName="kubernetes.io/configmap/e510aafb-e0d8-4898-a6ac-6aaaac776987-serviceca" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304391    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/projected/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-kube-api-access-p76fb" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304406    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd48639e-7f2c-4383-9716-29c1b4e68994" volumeName="kubernetes.io/projected/cd48639e-7f2c-4383-9716-29c1b4e68994-kube-api-access-pm4sb" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304418    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0808db1b-d15b-47c9-aa62-8a1a6309398b" volumeName="kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304431    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304447    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd48639e-7f2c-4383-9716-29c1b4e68994" volumeName="kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304459    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304473    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304492    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304506    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f1b72b8-3c48-4426-b83c-94bc84b69bf4" volumeName="kubernetes.io/configmap/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304518    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304533    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304546    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47825bca-e2dc-480b-b29e-3e071b260fd9" volumeName="kubernetes.io/projected/47825bca-e2dc-480b-b29e-3e071b260fd9-kube-api-access-5dwsx" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304558    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304573    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e510aafb-e0d8-4898-a6ac-6aaaac776987" volumeName="kubernetes.io/projected/e510aafb-e0d8-4898-a6ac-6aaaac776987-kube-api-access-hfml5" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304604    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304617    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47825bca-e2dc-480b-b29e-3e071b260fd9" volumeName="kubernetes.io/secret/47825bca-e2dc-480b-b29e-3e071b260fd9-metrics-tls" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304632    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" volumeName="kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304645    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c00de-9f0d-49c7-8cd8-37c99b9af0a7" volumeName="kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304657    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304669    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304682    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfd92d06-4598-48c4-822b-d0dff1df2936" volumeName="kubernetes.io/projected/cfd92d06-4598-48c4-822b-d0dff1df2936-kube-api-access-6crfc" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304695    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304707    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304720    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8924fe8-5503-4a1d-a361-18221c829534" volumeName="kubernetes.io/configmap/b8924fe8-5503-4a1d-a361-18221c829534-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304733    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" volumeName="kubernetes.io/projected/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-kube-api-access-69nwv" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304746    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" volumeName="kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304758    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf611fab-4203-425e-8db6-6ab6072ec4c2" volumeName="kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304775    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57299938-b4d9-45b6-9650-e0f0c9b50c80" volumeName="kubernetes.io/projected/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-api-access-7rkhk" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304793    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304812    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd48639e-7f2c-4383-9716-29c1b4e68994" volumeName="kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304825    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf611fab-4203-425e-8db6-6ab6072ec4c2" volumeName="kubernetes.io/projected/cf611fab-4203-425e-8db6-6ab6072ec4c2-kube-api-access-5kk45" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304840    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fca86dc7-9dba-44fa-af4b-209629079b49" volumeName="kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304858    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549292be-cfde-49a0-bd85-b1854a005bff" volumeName="kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304873    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f1b72b8-3c48-4426-b83c-94bc84b69bf4" volumeName="kubernetes.io/secret/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-agent-certs" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304891    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304905    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304918    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304934    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8924fe8-5503-4a1d-a361-18221c829534" volumeName="kubernetes.io/projected/b8924fe8-5503-4a1d-a361-18221c829534-kube-api-access-cb8ch" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304950    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" volumeName="kubernetes.io/empty-dir/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-textfile" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304967    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf611fab-4203-425e-8db6-6ab6072ec4c2" volumeName="kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304978    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0808db1b-d15b-47c9-aa62-8a1a6309398b" volumeName="kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.304991    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0808db1b-d15b-47c9-aa62-8a1a6309398b" volumeName="kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-env-overrides" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305017    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="522412d0-7818-4a3d-975f-c816eb5d7a13" volumeName="kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305035    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9fb057-17de-4446-a7cf-0730aaa6c46b" volumeName="kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305049    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc0a19a4-6a53-4252-8deb-30327d884859" volumeName="kubernetes.io/secret/cc0a19a4-6a53-4252-8deb-30327d884859-cert" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305063    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/configmap/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305076    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e30ed0-eb80-40bc-87bb-1afbdda8f1e3" volumeName="kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305091    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84aa625c-734a-4009-b534-c1acc4c247ee" volumeName="kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305104    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01662703-d552-4ae9-bc54-9e7da3c21bf3" volumeName="kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305139    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfd92d06-4598-48c4-822b-d0dff1df2936" volumeName="kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305151    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f768f332-b6fe-48bb-8e63-ea57b00f0b7d" volumeName="kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305164    5972 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48d98419-9d48-4a09-96cd-940bee92d0b3" volumeName="kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-tmp" seLinuxMountContext=""
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305177    5972 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:39:10.305975 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.305183    5972 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:39:10.331410 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.331378    5972 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:39:10.333202 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.333174    5972 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:39:10.333202 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.333201    5972 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:39:10.333414 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.333222    5972 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:39:10.333414 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.333264    5972 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:39:10.346738 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.346706    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:39:10.352488 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.352459    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:39:10.357757 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.357723    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550" exitCode=0
Oct 30 18:39:10.365007 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.364978    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e" exitCode=0
Oct 30 18:39:10.376019 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.375991    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:39:10.376386 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.376354    5972 generic.go:356] "Generic (PLEG): container finished" podID="0808db1b-d15b-47c9-aa62-8a1a6309398b" containerID="966f69808562e64c0d708009508f22e9e2658ddf29afacc9586f8414ed22f915" exitCode=1
Oct 30 18:39:10.389868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.389826    5972 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="6ffacf4bedca948ca2e8844f14a44aad8e90cdb7591b6778f8c2d710247b3fb0" exitCode=0
Oct 30 18:39:10.389868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.389853    5972 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="46ba7028298c4cd01441e4abbc7d3aa961658d2940c5af9f4f6ee153b74706c5" exitCode=0
Oct 30 18:39:10.389868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.389860    5972 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="c4ba4f3c070d8a055649157f1bbfe3a7f7346abdf729d3333a537a2d9c04982b" exitCode=0
Oct 30 18:39:10.389868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.389866    5972 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="f5810a664575fe0a0d78c0e7dc76b23cd052ad9e2a39f0826ab1215c08721017" exitCode=0
Oct 30 18:39:10.389868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.389877    5972 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="2333721f3f1cba4c94883f24dfb5c0b8e8f71fc944833151154a5ac3b0698cc9" exitCode=0
Oct 30 18:39:10.390170 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.389886    5972 generic.go:356] "Generic (PLEG): container finished" podID="cd48639e-7f2c-4383-9716-29c1b4e68994" containerID="eaa9e681a52fcee43eb53da434c4c79d525c899eede6c276bb77736c67b1492a" exitCode=0
Oct 30 18:39:10.397440 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.397412    5972 generic.go:356] "Generic (PLEG): container finished" podID="d47490914e2d482394d4603a5e95f123" containerID="f405a5e8a4c94710d5ff91502998311e9598fa2010745d2e9bf5b46487ff8469" exitCode=0
Oct 30 18:39:10.399890 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.399865    5972 generic.go:356] "Generic (PLEG): container finished" podID="bfbb88e7-35a1-4d40-9365-66a6ba00adb9" containerID="5fac431e9713b1bde31eb0800d333edbd22e9fc4e3b34b361c1e7ce628c7ef25" exitCode=0
Oct 30 18:39:10.400012 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.399956    5972 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:39:10.433443 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.433416    5972 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 18:39:10.438945 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.438924    5972 manager.go:324] Recovery completed
Oct 30 18:39:10.463420 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.463395    5972 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:39:10.465470 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.465435    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:39:10.465566 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.465477    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:39:10.465566 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.465492    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:39:10.468397 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468377    5972 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:39:10.468397 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468392    5972 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:39:10.468506 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468392    5972 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:39:10.468506 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468460    5972 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:39:10.468644 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468628    5972 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:39:10.468686 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468644    5972 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:39:10.468686 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468660    5972 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:39:10.468686 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468666    5972 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:39:10.468686 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468672    5972 policy_none.go:50] "Start"
Oct 30 18:39:10.468686 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468683    5972 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:39:10.468876 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468692    5972 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:39:10.468876 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468817    5972 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:39:10.468876 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468824    5972 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:39:10.468876 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.468830    5972 policy_none.go:44] "Start"
Oct 30 18:39:10.469367 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.469352    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:39:10.469452 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.469384    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:39:10.469452 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.469398    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:39:10.473641 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.473602    5972 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:39:10.477577 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.477564    5972 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:39:10.477657 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.477615    5972 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:39:10.477657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.477624    5972 server.go:87] "Starting device plugin registration server"
Oct 30 18:39:10.477886 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.477876    5972 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:39:10.477917 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.477887    5972 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:39:10.477980 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.477962    5972 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:39:10.478096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.478069    5972 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:39:10.478156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.478105    5972 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:39:10.478156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.478112    5972 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:39:10.478240 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.478165    5972 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:39:10.478095279Z","UUID":"36d33562-aa63-4b40-9769-65be4549c101","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:39:10.479790 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.479768    5972 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:39:10.479884 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.479813    5972 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-133.us-west-2.compute.internal\" not found"
Oct 30 18:39:10.480193 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.480175    5972 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:39:10.480272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.480211    5972 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:39:10.488032 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.488014    5972 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:10.488167 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.488153    5972 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:39:10.488899 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.488864    5972 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:10.578112 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.578033    5972 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:39:10.583624 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.582971    5972 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:10.583818 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.583800    5972 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:39:10.584357 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.584339    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:39:10.584411 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.584371    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:39:10.584411 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.584381    5972 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-133.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:39:10.584469 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.584423    5972 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.633635 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.633563    5972 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"]
Oct 30 18:39:10.634299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634283    5972 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.634353 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634306    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"c6d9ef131d394299abae730c3d467d74","Type":"ContainerStarted","Data":"3e2047a7fd4434e1f7b0a711acf83a078f27f4331b0b7a7863395e42afa5d2a5"}
Oct 30 18:39:10.634384 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634352    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"c6d9ef131d394299abae730c3d467d74","Type":"ContainerStarted","Data":"3a6f714126dc8aa2aeccb39658a3f10538dcb4f369b129594d7ba43a85889c55"}
Oct 30 18:39:10.634384 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634371    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"d47490914e2d482394d4603a5e95f123","Type":"ContainerStarted","Data":"26441fee0587bab0871583a948d68b2bc25417b7036fc98ed8a727ce98dbe0fa"}
Oct 30 18:39:10.634438 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634388    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"d47490914e2d482394d4603a5e95f123","Type":"ContainerDied","Data":"f405a5e8a4c94710d5ff91502998311e9598fa2010745d2e9bf5b46487ff8469"}
Oct 30 18:39:10.634438 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634406    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal" event={"ID":"d47490914e2d482394d4603a5e95f123","Type":"ContainerStarted","Data":"f00bad2d9d4bf7551b12a404f757cc7e40cbcef2cc45757b980fbaa4bb5183c3"}
Oct 30 18:39:10.634571 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.634561    5972 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.694319 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.694294    5972 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.694434 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.694359    5972 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.695108 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.695087    5972 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:10.695201 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.695139    5972 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.697951 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.697937    5972 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:10.698001 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:10.697974    5972 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.789318 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.789292    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6d9ef131d394299abae730c3d467d74-config\") pod \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"c6d9ef131d394299abae730c3d467d74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.789318 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.789318    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.789445 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.789334    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.890083 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.890012    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.890083 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.890043    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.890083 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.890060    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6d9ef131d394299abae730c3d467d74-config\") pod \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"c6d9ef131d394299abae730c3d467d74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.890256 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.890114    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6d9ef131d394299abae730c3d467d74-config\") pod \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"c6d9ef131d394299abae730c3d467d74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.890256 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.890117    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:10.890256 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:10.890151    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d47490914e2d482394d4603a5e95f123-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" (UID: \"d47490914e2d482394d4603a5e95f123\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:11.283959 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.283930    5972 apiserver.go:52] "Watching apiserver"
Oct 30 18:39:11.384255 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.384226    5972 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:39:11.384993 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.384968    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-rvg7d","openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch","openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m","openshift-multus/network-metrics-daemon-qd6bc","openshift-monitoring/alertmanager-main-0","openshift-ingress-canary/ingress-canary-55t9g","openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc","openshift-monitoring/metrics-server-58f458b77f-264xl","kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4md92","openshift-dns/dns-default-kz4df","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-fdf4b","openshift-multus/multus-n4t4x","openshift-dns/node-resolver-s4ps2","openshift-network-diagnostics/network-check-target-mdb7t","kube-system/global-pull-secret-syncer-77qht","openshift-image-registry/node-ca-7vv9s","openshift-monitoring/node-exporter-49tz6","openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl","openshift-multus/multus-additional-cni-plugins-4ljqw","openshift-monitoring/thanos-querier-5c948665c7-79g5z","openshift-ovn-kubernetes/ovnkube-node-shstn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal","openshift-insights/insights-runtime-extractor-pk298"]
Oct 30 18:39:11.402287 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.402262    5972 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:39:11.417216 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.415476    5972 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:11.417216 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.415890    5972 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:11.435918 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.435889    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.436043 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.435943    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.436127 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.436112    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27m54\""
Oct 30 18:39:11.436194 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.436114    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dxlbz\""
Oct 30 18:39:11.436244 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.436230    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6x86\""
Oct 30 18:39:11.436344 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.436332    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.436565 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.436554    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:39:11.436902 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.436888    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.437075 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437061    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:39:11.437147 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437136    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:39:11.437481 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437470    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.437618 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437593    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:39:11.437618 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437605    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.437772 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437637    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.437772 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.437669    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.438519 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.438504    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:39:11.438615 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.438558    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:39:11.438615 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.438568    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:39:11.438678 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.438558    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.438678 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.438617    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:39:11.438766 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.438752    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.439086 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439074    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swx9z\""
Oct 30 18:39:11.439257 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439241    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:39:11.439257 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439251    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:39:11.439339 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439249    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rrn4s\""
Oct 30 18:39:11.439339 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439279    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w46g4\""
Oct 30 18:39:11.439339 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439330    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4gd2\""
Oct 30 18:39:11.439435 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.439379    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:39:11.483488 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.483466    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.483488 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.483482    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:39:11.483762 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.483751    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 18:39:11.484845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484822    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 18:39:11.484934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484826    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 18:39:11.484934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484877    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.484934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484888    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.484934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484910    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 18:39:11.484934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484918    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.485105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484936    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:39:11.485105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.484876    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 18:39:11.485105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.485059    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:39:11.485105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.485077    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.485105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.485095    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:39:11.485243 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.485122    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.485636 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.485624    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.485697 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.485683    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.486196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.486159    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:39:11.486196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.486177    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.486317 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.486200    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.487433 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.487419    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 18:39:11.494702 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494684    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.494784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494709    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-tls\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.494784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494728    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-etc-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.494784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494749    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-socket-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.494891 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494838    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1fea5aa-6e82-4768-8f35-fb947216fb9f-hosts-file\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.494891 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494873    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-socket-dir-parent\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.494891 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494885    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495012 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494902    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-textfile\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.495012 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494970    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-kubelet\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.495012 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.494985    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-textfile\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.495145 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495018    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.495145 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495035    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-grpc-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.495145 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495060    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495152    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-var-lib-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.495293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495192    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.495293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495219    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495247    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495274    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-daemon-config\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495307    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-multus-certs\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495338    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crfc\" (UniqueName: \"kubernetes.io/projected/cfd92d06-4598-48c4-822b-d0dff1df2936-kube-api-access-6crfc\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495372    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495397    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-systemd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495422    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-node-log\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495448    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495462    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-daemon-config\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495477    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495531 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495505    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495532    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-system-cni-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495594    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495643    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495666    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47825bca-e2dc-480b-b29e-3e071b260fd9-config-volume\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495692    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495710    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495718    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495727    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495749    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495778    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495802    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtft\" (UniqueName: \"kubernetes.io/projected/0808db1b-d15b-47c9-aa62-8a1a6309398b-kube-api-access-xhtft\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495825    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47825bca-e2dc-480b-b29e-3e071b260fd9-config-volume\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495829    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-kubernetes\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495856    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495882    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-tls\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.495925 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495930    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495929    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495974    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fea5aa-6e82-4768-8f35-fb947216fb9f-tmp-dir\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495991    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8924fe8-5503-4a1d-a361-18221c829534-host-slash\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.495996    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496010    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-bin\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496036    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-env-overrides\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496087    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496125    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-tmp\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496140    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fea5aa-6e82-4768-8f35-fb947216fb9f-tmp-dir\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496162    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-env-overrides\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496199    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496213    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-tmp\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496232    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496268    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496327    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-modprobe-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496352    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496398    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-sys\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496405    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496424    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-netns\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496455    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496490    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47825bca-e2dc-480b-b29e-3e071b260fd9-metrics-tls\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496517    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496525    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk45\" (UniqueName: \"kubernetes.io/projected/cf611fab-4203-425e-8db6-6ab6072ec4c2-kube-api-access-5kk45\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.496657 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496566    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496687    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e510aafb-e0d8-4898-a6ac-6aaaac776987-serviceca\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496715    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9f7t\" (UniqueName: \"kubernetes.io/projected/522412d0-7818-4a3d-975f-c816eb5d7a13-kube-api-access-b9f7t\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496743    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-metrics-server-audit-profiles\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496768    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-agent-certs\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496796    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt5k\" (UniqueName: \"kubernetes.io/projected/48d98419-9d48-4a09-96cd-940bee92d0b3-kube-api-access-2tt5k\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496825    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-device-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496880    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.496903    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbn8\" (UniqueName: \"kubernetes.io/projected/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kube-api-access-crbn8\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497078    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497290    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-federate-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497331    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-metrics-client-ca\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497377    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-netns\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497381    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-agent-certs\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497415    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovn-node-metrics-cert\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497451    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-audit-log\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497494    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8924fe8-5503-4a1d-a361-18221c829534-iptables-alerter-script\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497524    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysconfig\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497558    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-sys\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497610    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-sys-fs\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.497616 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497625    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-audit-log\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497645    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkhk\" (UniqueName: \"kubernetes.io/projected/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-api-access-7rkhk\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497725    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e510aafb-e0d8-4898-a6ac-6aaaac776987-serviceca\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497865    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-metrics-server-audit-profiles\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498005    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovn-node-metrics-cert\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.497686    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498075    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-dbus\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498122    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-ovn\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498216    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498222    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-metrics-client-ca\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498253    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498350    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498391    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:11.498462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498454    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-client-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498489    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7brm\" (UniqueName: \"kubernetes.io/projected/f1fea5aa-6e82-4768-8f35-fb947216fb9f-kube-api-access-j7brm\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498523    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-conf\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498560    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498619    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-k8s-cni-cncf-io\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498650    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498682    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf611fab-4203-425e-8db6-6ab6072ec4c2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498716    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfml5\" (UniqueName: \"kubernetes.io/projected/e510aafb-e0d8-4898-a6ac-6aaaac776987-kube-api-access-hfml5\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498749    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/522412d0-7818-4a3d-975f-c816eb5d7a13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498776    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498812    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498819    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca86dc7-9dba-44fa-af4b-209629079b49-metrics-certs\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:11.499028 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.498844    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499069    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499124    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-log-socket\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499156    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499199    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8ch\" (UniqueName: \"kubernetes.io/projected/b8924fe8-5503-4a1d-a361-18221c829534-kube-api-access-cb8ch\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499234    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499264    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.499560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499471    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.499889 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499671    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf611fab-4203-425e-8db6-6ab6072ec4c2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.499889 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499679    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/522412d0-7818-4a3d-975f-c816eb5d7a13-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.499889 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499738    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.499889 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499790    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.499889 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499831    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-var-lib-kubelet\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.499889 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499859    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.500173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.499892    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.500173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500031    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.500173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500072    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-cnibin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.500173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500101    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-hostroot\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.500173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500140    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-conf-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.500173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500167    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-wtmp\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500195    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-script-lib\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500233    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-client-certs\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500262    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500306    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500333    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-multus\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500369    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-etc-kubernetes\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.500413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500401    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nwv\" (UniqueName: \"kubernetes.io/projected/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-kube-api-access-69nwv\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.500724 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500569    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-systemd\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.500724 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500626    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-tuned\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.500724 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500659    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwsx\" (UniqueName: \"kubernetes.io/projected/47825bca-e2dc-480b-b29e-3e071b260fd9-kube-api-access-5dwsx\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.500724 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500701    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76fb\" (UniqueName: \"kubernetes.io/projected/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-kube-api-access-p76fb\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.500894 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500754    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57299938-b4d9-45b6-9650-e0f0c9b50c80-data-volume\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.500894 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500786    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.500894 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500817    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-konnectivity-ca\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:39:11.500894 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500858    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.501058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500892    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57299938-b4d9-45b6-9650-e0f0c9b50c80-crio-socket\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.501058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500937    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-cnibin\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.501058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.500970    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.501058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501012    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-netd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.501058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501037    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-tmp\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501066    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501098    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501127    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501166    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-kubelet-config\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501198    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrcr\" (UniqueName: \"kubernetes.io/projected/549292be-cfde-49a0-bd85-b1854a005bff-kube-api-access-vwrcr\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501229    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-root\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.501264 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501261    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47825bca-e2dc-480b-b29e-3e071b260fd9-tmp-dir\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.501551 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501292    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.501551 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501319    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-metrics-client-ca\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.501551 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501362    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.501551 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501395    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-cni-binary-copy\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.501797 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501704    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-kubelet\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.501797 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501761    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-config\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.501797 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501792    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.501922 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501825    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-run\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.501922 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501861    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.501922 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501889    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-kube-api-access-7tnhb\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.502049 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501919    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-system-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.502049 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501949    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-slash\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.502049 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.501977    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.502049 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502004    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-host\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.502049 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502028    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.502258 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502059    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.502258 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502090    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.502258 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502121    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-systemd-units\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.502403 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502265    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-tmp\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.502403 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502372    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.502494 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502405    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47825bca-e2dc-480b-b29e-3e071b260fd9-tmp-dir\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.502494 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502427    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.502494 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502434    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-config\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.502708 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502689    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.502754 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502735    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.502793 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502746    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrhq\" (UniqueName: \"kubernetes.io/projected/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-kube-api-access-xwrhq\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.502793 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502751    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.502793 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.502773    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0808db1b-d15b-47c9-aa62-8a1a6309398b-ovnkube-script-lib\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503142    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503207    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503256    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-lib-modules\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503262    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd92d06-4598-48c4-822b-d0dff1df2936-cni-binary-copy\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503274    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-tuned\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503279    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f1b72b8-3c48-4426-b83c-94bc84b69bf4-konnectivity-ca\") pod \"konnectivity-agent-fdf4b\" (UID: \"5f1b72b8-3c48-4426-b83c-94bc84b69bf4\") " pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:39:11.503315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503312    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.503714 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503367    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.503714 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503407    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.503714 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503466    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/549292be-cfde-49a0-bd85-b1854a005bff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.503714 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503504    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-accelerators-collector-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.503714 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503537    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.503714 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503630    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/549292be-cfde-49a0-bd85-b1854a005bff-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.503980 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503787    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57299938-b4d9-45b6-9650-e0f0c9b50c80-data-volume\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.503980 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503798    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd48639e-7f2c-4383-9716-29c1b4e68994-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.503980 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503825    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-accelerators-collector-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.503980 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503845    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-metrics-client-ca\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.503980 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503877    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4sb\" (UniqueName: \"kubernetes.io/projected/cd48639e-7f2c-4383-9716-29c1b4e68994-kube-api-access-pm4sb\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.504127 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.503991    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e510aafb-e0d8-4898-a6ac-6aaaac776987-host\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.504127 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504093    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-registration-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.504216 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504143    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.504216 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504199    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-metrics-client-ca\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.504298 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504235    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xtl\" (UniqueName: \"kubernetes.io/projected/fca86dc7-9dba-44fa-af4b-209629079b49-kube-api-access-26xtl\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:11.504354 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504339    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57299938-b4d9-45b6-9650-e0f0c9b50c80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.504409 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504385    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-metrics-client-ca\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.504457 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504432    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-bin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.504508 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504462    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0a19a4-6a53-4252-8deb-30327d884859-cert\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:39:11.504508 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504500    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4gz\" (UniqueName: \"kubernetes.io/projected/cc0a19a4-6a53-4252-8deb-30327d884859-kube-api-access-pm4gz\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:39:11.504670 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504562    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.504705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.504683    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.532912 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.532891    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:39:11.533003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.532932    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:39:11.533914 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.533891    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:39:11.534034 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.533945    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 18:39:11.534034 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.533969    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 18:39:11.534034 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534006    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:39:11.534034 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534004    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 18:39:11.534355 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534238    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:39:11.534355 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534256    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:39:11.534355 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534260    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4mf4w\""
Oct 30 18:39:11.534534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534424    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlkjn\""
Oct 30 18:39:11.534534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534426    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:39:11.534534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534438    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:39:11.534534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534464    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vpq5z\""
Oct 30 18:39:11.534534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534497    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534546    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534556    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5c5lj\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534642    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvdpg\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534671    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534671    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534704    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534723    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:39:11.534764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534738    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534741    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534797    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gg474\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534824    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-64dnq\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534849    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534939    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gkr95\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534956    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jxx4c\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.534983    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fp92n78v94iu5\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535030    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535076    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l87lm\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535082    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535112    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:39:11.535122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535116    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535077    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535139    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535153    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wrflg\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535162    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535185    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535203    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535223    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535230    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535258    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535322    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535373    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-86nv8\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535418    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535430    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535424    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535441    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535462    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6fhn6\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535536    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535554    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535557    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akuufua83pjtp\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535572    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535620    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535622    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535641    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535624    5972 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535671    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535674    5972 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535623    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535734    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 18:39:11.535758 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:11.535734    5972 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535807    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535861    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535870    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535890    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535941    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1bcilccoormb6\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535956    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.535972    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536017    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536087    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w6qhz\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536106    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhcv2\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536133    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-tls\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536186    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kgr5v\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536286    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zrdzz\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:11.536325    5972 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-133.us-west-2.compute.internal"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536356    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536484    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536490    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536639    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536668    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-original-pull-secret\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.536820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.536784    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47825bca-e2dc-480b-b29e-3e071b260fd9-metrics-tls\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.537682 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.537667    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-federate-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.538410 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538395    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.538476 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538413    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.538476 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538434    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8924fe8-5503-4a1d-a361-18221c829534-iptables-alerter-script\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.538595 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538497    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.538639 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538570    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.538671 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538657    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.538713 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.538661    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.539536 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.539519    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.539612 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.539595    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf611fab-4203-425e-8db6-6ab6072ec4c2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.539659 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.539600    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-client-ca-bundle\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.542734 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.542704    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.542817 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.542743    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.542817 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.542749    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.542817 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.542796    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-secret-metrics-server-client-certs\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.542908 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.542843    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.543991 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.543907    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.543991 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.543931    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-secret-telemeter-client\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.543991 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.543951    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/549292be-cfde-49a0-bd85-b1854a005bff-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.543991 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.543954    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.543991 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.543977    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.544299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544011    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.544299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544035    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.544299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544050    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.544299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544073    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.544299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544115    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.544299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544201    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.544485 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.544299    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.545360 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545339    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.545360 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545355    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0a19a4-6a53-4252-8deb-30327d884859-cert\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:39:11.545474 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545399    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57299938-b4d9-45b6-9650-e0f0c9b50c80-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.545474 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545426    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-secret-grpc-tls\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.545552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545486    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.545552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545527    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.545552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545547    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-tls\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.545653 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545611    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.545771 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.545759    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-telemeter-client-tls\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.546728 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.546714    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/522412d0-7818-4a3d-975f-c816eb5d7a13-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.552000 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.551985    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crfc\" (UniqueName: \"kubernetes.io/projected/cfd92d06-4598-48c4-822b-d0dff1df2936-kube-api-access-6crfc\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.553803 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.553785    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk45\" (UniqueName: \"kubernetes.io/projected/cf611fab-4203-425e-8db6-6ab6072ec4c2-kube-api-access-5kk45\") pod \"openshift-state-metrics-7bd7489957-gr5bl\" (UID: \"cf611fab-4203-425e-8db6-6ab6072ec4c2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gr5bl"
Oct 30 18:39:11.553984 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.553963    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/9a9fb057-17de-4446-a7cf-0730aaa6c46b-kube-api-access-jn4cw\") pod \"network-check-target-mdb7t\" (UID: \"9a9fb057-17de-4446-a7cf-0730aaa6c46b\") " pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:11.554117 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.554047    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") pod \"prometheus-k8s-0\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.554175 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.554116    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9f7t\" (UniqueName: \"kubernetes.io/projected/522412d0-7818-4a3d-975f-c816eb5d7a13-kube-api-access-b9f7t\") pod \"prometheus-operator-9cd89b74c-mpc5m\" (UID: \"522412d0-7818-4a3d-975f-c816eb5d7a13\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mpc5m"
Oct 30 18:39:11.554227 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.554182    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtft\" (UniqueName: \"kubernetes.io/projected/0808db1b-d15b-47c9-aa62-8a1a6309398b-kube-api-access-xhtft\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.554227 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.554206    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkhk\" (UniqueName: \"kubernetes.io/projected/57299938-b4d9-45b6-9650-e0f0c9b50c80-kube-api-access-7rkhk\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.554227 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.554213    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbn8\" (UniqueName: \"kubernetes.io/projected/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kube-api-access-crbn8\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.554454 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.554434    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt5k\" (UniqueName: \"kubernetes.io/projected/48d98419-9d48-4a09-96cd-940bee92d0b3-kube-api-access-2tt5k\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.555548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.555532    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7brm\" (UniqueName: \"kubernetes.io/projected/f1fea5aa-6e82-4768-8f35-fb947216fb9f-kube-api-access-j7brm\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.561022 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.561004    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") pod \"alertmanager-main-0\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:11.561118 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.561031    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrhq\" (UniqueName: \"kubernetes.io/projected/65e30ed0-eb80-40bc-87bb-1afbdda8f1e3-kube-api-access-xwrhq\") pod \"thanos-querier-5c948665c7-79g5z\" (UID: \"65e30ed0-eb80-40bc-87bb-1afbdda8f1e3\") " pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:11.561118 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.561100    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrcr\" (UniqueName: \"kubernetes.io/projected/549292be-cfde-49a0-bd85-b1854a005bff-kube-api-access-vwrcr\") pod \"kube-state-metrics-5dc8d7d89b-r6mch\" (UID: \"549292be-cfde-49a0-bd85-b1854a005bff\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r6mch"
Oct 30 18:39:11.562936 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.562920    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwsx\" (UniqueName: \"kubernetes.io/projected/47825bca-e2dc-480b-b29e-3e071b260fd9-kube-api-access-5dwsx\") pod \"dns-default-kz4df\" (UID: \"47825bca-e2dc-480b-b29e-3e071b260fd9\") " pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:11.563072 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563057    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/166c00de-9f0d-49c7-8cd8-37c99b9af0a7-kube-api-access-7tnhb\") pod \"metrics-server-58f458b77f-264xl\" (UID: \"166c00de-9f0d-49c7-8cd8-37c99b9af0a7\") " pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:11.563251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563232    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfml5\" (UniqueName: \"kubernetes.io/projected/e510aafb-e0d8-4898-a6ac-6aaaac776987-kube-api-access-hfml5\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.563647 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563628    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xtl\" (UniqueName: \"kubernetes.io/projected/fca86dc7-9dba-44fa-af4b-209629079b49-kube-api-access-26xtl\") pod \"network-metrics-daemon-qd6bc\" (UID: \"fca86dc7-9dba-44fa-af4b-209629079b49\") " pod="openshift-multus/network-metrics-daemon-qd6bc"
Oct 30 18:39:11.563734 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563665    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nwv\" (UniqueName: \"kubernetes.io/projected/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-kube-api-access-69nwv\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.563784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563733    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4sb\" (UniqueName: \"kubernetes.io/projected/cd48639e-7f2c-4383-9716-29c1b4e68994-kube-api-access-pm4sb\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.563784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563764    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8ch\" (UniqueName: \"kubernetes.io/projected/b8924fe8-5503-4a1d-a361-18221c829534-kube-api-access-cb8ch\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.563932 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563913    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4gz\" (UniqueName: \"kubernetes.io/projected/cc0a19a4-6a53-4252-8deb-30327d884859-kube-api-access-pm4gz\") pod \"ingress-canary-55t9g\" (UID: \"cc0a19a4-6a53-4252-8deb-30327d884859\") " pod="openshift-ingress-canary/ingress-canary-55t9g"
Oct 30 18:39:11.564000 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.563987    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76fb\" (UniqueName: \"kubernetes.io/projected/f768f332-b6fe-48bb-8e63-ea57b00f0b7d-kube-api-access-p76fb\") pod \"telemeter-client-7f9b8d4dbf-wn9xw\" (UID: \"f768f332-b6fe-48bb-8e63-ea57b00f0b7d\") " pod="openshift-monitoring/telemeter-client-7f9b8d4dbf-wn9xw"
Oct 30 18:39:11.605815 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605792    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.605887 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605819    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e510aafb-e0d8-4898-a6ac-6aaaac776987-host\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.605887 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605835    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-registration-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.605887 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605862    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-bin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605891    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-etc-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605892    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e510aafb-e0d8-4898-a6ac-6aaaac776987-host\") pod \"node-ca-7vv9s\" (UID: \"e510aafb-e0d8-4898-a6ac-6aaaac776987\") " pod="openshift-image-registry/node-ca-7vv9s"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605907    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605915    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-socket-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605917    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-registration-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605937    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-etc-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605936    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-bin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605947    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1fea5aa-6e82-4768-8f35-fb947216fb9f-hosts-file\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605964    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-socket-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605988    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1fea5aa-6e82-4768-8f35-fb947216fb9f-hosts-file\") pod \"node-resolver-s4ps2\" (UID: \"f1fea5aa-6e82-4768-8f35-fb947216fb9f\") " pod="openshift-dns/node-resolver-s4ps2"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.605989    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-socket-dir-parent\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606015    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-kubelet\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606032    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-var-lib-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606036    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-socket-dir-parent\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606058    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-multus-certs\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606074    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-kubelet\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606079    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-var-lib-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606085    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-systemd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606109    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-node-log\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606114    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-systemd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606119    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-multus-certs\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606136    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-system-cni-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606148    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-node-log\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606155    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-kubernetes\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606171    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8924fe8-5503-4a1d-a361-18221c829534-host-slash\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606191    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-bin\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606217    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-modprobe-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606226    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-kubernetes\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606234    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8924fe8-5503-4a1d-a361-18221c829534-host-slash\") pod \"iptables-alerter-rvg7d\" (UID: \"b8924fe8-5503-4a1d-a361-18221c829534\") " pod="openshift-network-operator/iptables-alerter-rvg7d"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606241    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-sys\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606239    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-system-cni-dir\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606252    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-bin\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606277    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-sys\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606307    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-netns\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606329    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-modprobe-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606341    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-device-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606358    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-netns\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606370    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-netns\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606398    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysconfig\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606409    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-device-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606421    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-sys\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606446    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-sys-fs\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606465    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysconfig\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606497    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-sys\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606497    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-netns\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606517    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-dbus\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606542    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-ovn\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606502    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-sys-fs\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606568    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-ovn\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606574    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-conf\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606601    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-dbus\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606626    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-k8s-cni-cncf-io\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606645    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-log-socket\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606661    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606679    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-run-k8s-cni-cncf-io\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606684    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.606681 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606695    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-log-socket\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606714    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606721    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606725    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606739    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-var-lib-kubelet\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606762    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-cnibin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606772    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-conf\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606778    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-hostroot\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606799    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-conf-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606811    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-run-ovn-kubernetes\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606817    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-wtmp\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606819    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-cnibin\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606827    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-hostroot\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606841    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-multus\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606851    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-var-lib-kubelet\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606862    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-multus-conf-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606913    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-node-exporter-wtmp\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606921    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-etc-kubernetes\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606924    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-cni-multus\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606945    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-etc-kubernetes\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606958    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-systemd\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.606994    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57299938-b4d9-45b6-9650-e0f0c9b50c80-crio-socket\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607008    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-cnibin\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607023    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-netd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607040    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607058    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57299938-b4d9-45b6-9650-e0f0c9b50c80-crio-socket\") pod \"insights-runtime-extractor-pk298\" (UID: \"57299938-b4d9-45b6-9650-e0f0c9b50c80\") " pod="openshift-insights/insights-runtime-extractor-pk298"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607040    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-systemd\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607060    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd48639e-7f2c-4383-9716-29c1b4e68994-cnibin\") pod \"multus-additional-cni-plugins-4ljqw\" (UID: \"cd48639e-7f2c-4383-9716-29c1b4e68994\") " pod="openshift-multus/multus-additional-cni-plugins-4ljqw"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607072    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-cni-netd\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607062    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-kubelet-config\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607066    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607090    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b-kubelet-config\") pod \"global-pull-secret-syncer-77qht\" (UID: \"8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b\") " pod="kube-system/global-pull-secret-syncer-77qht"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607105    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-root\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607124    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-kubelet\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607140    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfbb88e7-35a1-4d40-9365-66a6ba00adb9-root\") pod \"node-exporter-49tz6\" (UID: \"bfbb88e7-35a1-4d40-9365-66a6ba00adb9\") " pod="openshift-monitoring/node-exporter-49tz6"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607156    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607170    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-run\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607185    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-system-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607200    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-slash\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607214    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607228    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-host\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607252    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-host-slash\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607253    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-systemd-units\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607272    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-run-openvswitch\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607230    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-run\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607255    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-host-var-lib-kubelet\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607287    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607297    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-host\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607305    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-etc-sysctl-d\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607315    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-lib-modules\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607319    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d87c914-b46f-4c4e-a7d9-3eb4335c463d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvwpc\" (UID: \"9d87c914-b46f-4c4e-a7d9-3eb4335c463d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvwpc"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607254    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd92d06-4598-48c4-822b-d0dff1df2936-system-cni-dir\") pod \"multus-n4t4x\" (UID: \"cfd92d06-4598-48c4-822b-d0dff1df2936\") " pod="openshift-multus/multus-n4t4x"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607337    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0808db1b-d15b-47c9-aa62-8a1a6309398b-systemd-units\") pod \"ovnkube-node-shstn\" (UID: \"0808db1b-d15b-47c9-aa62-8a1a6309398b\") " pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:11.608068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.607406    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48d98419-9d48-4a09-96cd-940bee92d0b3-lib-modules\") pod \"tuned-4md92\" (UID: \"48d98419-9d48-4a09-96cd-940bee92d0b3\") " pod="openshift-cluster-node-tuning-operator/tuned-4md92"
Oct 30 18:39:11.914830 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.914748    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:11.933552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:11.933527    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:12.181251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:12.181187    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:12.322632 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:12.322601    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kz4df"
Oct 30 18:39:12.607361 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:12.607325    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:14.989711 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:14.989680    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c948665c7-79g5z"
Oct 30 18:39:17.688828 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:17.688792    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:19.468810 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:19.468721    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:19.469319 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:19.468954    5972 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:39:21.339949 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.339915    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shstn"
Oct 30 18:39:21.993884 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.993849    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:21.994464 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.994432    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="alertmanager" containerID="cri-o://7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3" gracePeriod=120
Oct 30 18:39:21.994559 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.994506    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="prom-label-proxy" containerID="cri-o://0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e" gracePeriod=120
Oct 30 18:39:21.994705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.994540    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-web" containerID="cri-o://fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e" gracePeriod=120
Oct 30 18:39:21.994705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.994592    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy" containerID="cri-o://0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e" gracePeriod=120
Oct 30 18:39:21.994705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.994506    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-metric" containerID="cri-o://841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e" gracePeriod=120
Oct 30 18:39:21.994705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:21.994676    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="config-reloader" containerID="cri-o://a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86" gracePeriod=120
Oct 30 18:39:22.171699 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.171671    5972 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:22.464015 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.463986    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e" exitCode=0
Oct 30 18:39:22.464015 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464009    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e" exitCode=0
Oct 30 18:39:22.464015 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464015    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e" exitCode=0
Oct 30 18:39:22.464015 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464022    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86" exitCode=0
Oct 30 18:39:22.464462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464029    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3" exitCode=0
Oct 30 18:39:22.464462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464054    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e"}
Oct 30 18:39:22.464462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464115    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e"}
Oct 30 18:39:22.464462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464147    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e"}
Oct 30 18:39:22.464462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464175    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86"}
Oct 30 18:39:22.464462 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:22.464199    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3"}
Oct 30 18:39:22.869826 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:22.869743    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:39:23.243248 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.243225    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.321536 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321509    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321664 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321548    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321664 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321592    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321664 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321622    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321664 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321653    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321676    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321730    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321762    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321804    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.321845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321836    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.322098 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321884    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.322098 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321918    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.322098 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.321961    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") pod \"01662703-d552-4ae9-bc54-9e7da3c21bf3\" (UID: \"01662703-d552-4ae9-bc54-9e7da3c21bf3\") "
Oct 30 18:39:23.322733 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.322444    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:23.323208 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.322972    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:23.323971 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.323915    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:23.325560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.325433    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.325723 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.325685    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:23.326507 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.326484    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.326673 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.326565    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.326846 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.326820    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:23.326909 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.326840    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.327075 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.327057    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "kube-api-access-dmc9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:23.327232 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.327210    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.330162 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.330055    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.336860 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.336838    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config" pod "01662703-d552-4ae9-bc54-9e7da3c21bf3" (UID: "01662703-d552-4ae9-bc54-9e7da3c21bf3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:23.423279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423218    5972 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-cluster-tls-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423243    5972 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423253    5972 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-metrics-client-ca\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423262    5972 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-main-db\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423272    5972 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01662703-d552-4ae9-bc54-9e7da3c21bf3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423280    5972 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423291    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmc9l\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-kube-api-access-dmc9l\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423300    5972 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-main-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423309    5972 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01662703-d552-4ae9-bc54-9e7da3c21bf3-tls-assets\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423317    5972 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423327    5972 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-out\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423335    5972 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-config-volume\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.423543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.423344    5972 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01662703-d552-4ae9-bc54-9e7da3c21bf3-web-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:23.471786 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.471757    5972 generic.go:356] "Generic (PLEG): container finished" podID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerID="841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e" exitCode=0
Oct 30 18:39:23.472138 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.471830    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.472138 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.471835    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e"}
Oct 30 18:39:23.472138 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.471900    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01662703-d552-4ae9-bc54-9e7da3c21bf3","Type":"ContainerDied","Data":"56f9e82c7cd91286ca03208543ed4cfedac436289f311362e50e3b0d5bc7eadb"}
Oct 30 18:39:23.472138 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.471932    5972 scope.go:122] "RemoveContainer" containerID="0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e"
Oct 30 18:39:23.480978 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.480964    5972 scope.go:122] "RemoveContainer" containerID="841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e"
Oct 30 18:39:23.488472 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.488456    5972 scope.go:122] "RemoveContainer" containerID="0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e"
Oct 30 18:39:23.495485 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.495466    5972 scope.go:122] "RemoveContainer" containerID="fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e"
Oct 30 18:39:23.502281 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.502268    5972 scope.go:122] "RemoveContainer" containerID="a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86"
Oct 30 18:39:23.509275 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.509250    5972 scope.go:122] "RemoveContainer" containerID="7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3"
Oct 30 18:39:23.516614 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.516600    5972 scope.go:122] "RemoveContainer" containerID="e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e"
Oct 30 18:39:23.523543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.523525    5972 scope.go:122] "RemoveContainer" containerID="0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e"
Oct 30 18:39:23.523860 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.523837    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e\": container with ID starting with 0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e not found: ID does not exist" containerID="0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e"
Oct 30 18:39:23.523908 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.523867    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e"} err="failed to get container status \"0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e\": rpc error: code = NotFound desc = could not find container \"0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e\": container with ID starting with 0ea4a94a3f6dce194cd6e8707ab63bc5069c1febfbedc26345bac9cf1003c13e not found: ID does not exist"
Oct 30 18:39:23.523908 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.523897    5972 scope.go:122] "RemoveContainer" containerID="841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e"
Oct 30 18:39:23.524205 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.524189    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e\": container with ID starting with 841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e not found: ID does not exist" containerID="841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e"
Oct 30 18:39:23.524265 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524210    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e"} err="failed to get container status \"841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e\": rpc error: code = NotFound desc = could not find container \"841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e\": container with ID starting with 841f5653e552f4892b8ea4a5f88a66595f952ded86b1f781c421c31ae7921b3e not found: ID does not exist"
Oct 30 18:39:23.524265 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524226    5972 scope.go:122] "RemoveContainer" containerID="0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e"
Oct 30 18:39:23.524476 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.524459    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e\": container with ID starting with 0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e not found: ID does not exist" containerID="0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e"
Oct 30 18:39:23.524516 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524482    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e"} err="failed to get container status \"0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e\": rpc error: code = NotFound desc = could not find container \"0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e\": container with ID starting with 0a9c4785ec41a034260a474b9d72a6b65d337422329be737e72b4f23d7f4f19e not found: ID does not exist"
Oct 30 18:39:23.524516 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524502    5972 scope.go:122] "RemoveContainer" containerID="fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e"
Oct 30 18:39:23.524744 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.524729    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e\": container with ID starting with fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e not found: ID does not exist" containerID="fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e"
Oct 30 18:39:23.524813 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524751    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e"} err="failed to get container status \"fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e\": rpc error: code = NotFound desc = could not find container \"fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e\": container with ID starting with fc58a6402e13520728d2e649f98df6fd0d11934f29c9203d44cee7c91570129e not found: ID does not exist"
Oct 30 18:39:23.524813 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524764    5972 scope.go:122] "RemoveContainer" containerID="a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86"
Oct 30 18:39:23.524993 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.524979    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86\": container with ID starting with a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86 not found: ID does not exist" containerID="a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86"
Oct 30 18:39:23.525035 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.524996    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86"} err="failed to get container status \"a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86\": rpc error: code = NotFound desc = could not find container \"a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86\": container with ID starting with a92853409f89a6ac78492f3c91c7946788e195ea2ce97213ae42bff4f5e86b86 not found: ID does not exist"
Oct 30 18:39:23.525035 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.525009    5972 scope.go:122] "RemoveContainer" containerID="7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3"
Oct 30 18:39:23.525218 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.525201    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3\": container with ID starting with 7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3 not found: ID does not exist" containerID="7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3"
Oct 30 18:39:23.525267 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.525225    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3"} err="failed to get container status \"7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3\": rpc error: code = NotFound desc = could not find container \"7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3\": container with ID starting with 7a2bcb68d14d3198f7864b291a101a058e47f0b88f628e649f2a07e0bb113fb3 not found: ID does not exist"
Oct 30 18:39:23.525267 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.525244    5972 scope.go:122] "RemoveContainer" containerID="e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e"
Oct 30 18:39:23.525499 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:23.525484    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e\": container with ID starting with e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e not found: ID does not exist" containerID="e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e"
Oct 30 18:39:23.525543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.525502    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e"} err="failed to get container status \"e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e\": rpc error: code = NotFound desc = could not find container \"e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e\": container with ID starting with e8f3bfc3c550c97c68e4369d1d2af152b7527a7dde11b8f38ea4c848e4d3332e not found: ID does not exist"
Oct 30 18:39:23.594745 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.594716    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:23.604446 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.604417    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:23.643904 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.643875    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:23.644188 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644175    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="prom-label-proxy"
Oct 30 18:39:23.644229 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644189    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="prom-label-proxy"
Oct 30 18:39:23.644229 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644212    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy"
Oct 30 18:39:23.644229 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644218    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy"
Oct 30 18:39:23.644229 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644225    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="config-reloader"
Oct 30 18:39:23.644229 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644230    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="config-reloader"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644241    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="alertmanager"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644245    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="alertmanager"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644252    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="init-config-reloader"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644257    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="init-config-reloader"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644265    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-metric"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644270    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-metric"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644293    5972 cpu_manager.go:414] "containerMap: removing container" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-web"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644299    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-web"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644371    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644379    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="config-reloader"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644386    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="alertmanager"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644391    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="init-config-reloader"
Oct 30 18:39:23.644396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644397    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-metric"
Oct 30 18:39:23.644766 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644405    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="kube-rbac-proxy-web"
Oct 30 18:39:23.644766 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.644412    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" containerName="prom-label-proxy"
Oct 30 18:39:23.650473 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.650451    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.707721 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707698    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:39:23.707721 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707720    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:39:23.707903 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707701    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:39:23.707903 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707700    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:39:23.708009 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707929    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:39:23.708009 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707940    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:39:23.708009 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707944    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhcv2\""
Oct 30 18:39:23.708009 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.707951    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:39:23.708442 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.708419    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:39:23.725244 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725207    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725244 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725240    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725466 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725269    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008b7f60-84ef-44fb-a059-66ea1da81563-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725466 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725403    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008b7f60-84ef-44fb-a059-66ea1da81563-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725466 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725453    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008b7f60-84ef-44fb-a059-66ea1da81563-config-out\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725659 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725494    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snhj\" (UniqueName: \"kubernetes.io/projected/008b7f60-84ef-44fb-a059-66ea1da81563-kube-api-access-7snhj\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725659 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725525    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008b7f60-84ef-44fb-a059-66ea1da81563-tls-assets\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725659 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725598    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725659 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725633    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-config-volume\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725659 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725649    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725913 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725674    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-web-config\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725913 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725743    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.725913 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.725759    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008b7f60-84ef-44fb-a059-66ea1da81563-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.750885 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.750860    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:39:23.755966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.755948    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:23.826666 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826642    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008b7f60-84ef-44fb-a059-66ea1da81563-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.826829 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826677    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008b7f60-84ef-44fb-a059-66ea1da81563-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.826829 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826701    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008b7f60-84ef-44fb-a059-66ea1da81563-config-out\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.826829 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826717    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snhj\" (UniqueName: \"kubernetes.io/projected/008b7f60-84ef-44fb-a059-66ea1da81563-kube-api-access-7snhj\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.826979 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826844    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008b7f60-84ef-44fb-a059-66ea1da81563-tls-assets\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.826979 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826882    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.826979 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826961    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-config-volume\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.826989    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827014    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-web-config\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827060    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827085    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008b7f60-84ef-44fb-a059-66ea1da81563-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827119    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827382 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827149    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827382 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827142    5972 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:39:23.827540 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827515    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008b7f60-84ef-44fb-a059-66ea1da81563-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.827749 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.827693    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008b7f60-84ef-44fb-a059-66ea1da81563-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.828328 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.828302    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008b7f60-84ef-44fb-a059-66ea1da81563-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.830293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.830241    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008b7f60-84ef-44fb-a059-66ea1da81563-config-out\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.830405 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.830373    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.830916 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.830892    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.831025 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.830922    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.831025 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.830923    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.831130 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.831076    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.831228 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.831210    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008b7f60-84ef-44fb-a059-66ea1da81563-tls-assets\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.831262 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.831220    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-config-volume\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.832008 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.831986    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008b7f60-84ef-44fb-a059-66ea1da81563-web-config\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.885679 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.885658    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snhj\" (UniqueName: \"kubernetes.io/projected/008b7f60-84ef-44fb-a059-66ea1da81563-kube-api-access-7snhj\") pod \"alertmanager-main-0\" (UID: \"008b7f60-84ef-44fb-a059-66ea1da81563\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:23.980246 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:23.980183    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:39:24.338598 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:24.338510    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01662703-d552-4ae9-bc54-9e7da3c21bf3" path="/var/lib/kubelet/pods/01662703-d552-4ae9-bc54-9e7da3c21bf3/volumes"
Oct 30 18:39:24.406627 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:24.406597    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:39:24.410183 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:24.410157    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008b7f60_84ef_44fb_a059_66ea1da81563.slice/crio-d58f28f22dae12809117366eb17226ee281684313c6c8c73ca481fbe0d4df968 WatchSource:0}: Error finding container d58f28f22dae12809117366eb17226ee281684313c6c8c73ca481fbe0d4df968: Status 404 returned error can't find the container with id d58f28f22dae12809117366eb17226ee281684313c6c8c73ca481fbe0d4df968
Oct 30 18:39:24.478251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:24.478228    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"d58f28f22dae12809117366eb17226ee281684313c6c8c73ca481fbe0d4df968"}
Oct 30 18:39:25.483245 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:25.483217    5972 generic.go:356] "Generic (PLEG): container finished" podID="008b7f60-84ef-44fb-a059-66ea1da81563" containerID="9e36175dd7b3eeea81ad19150e4e8d9b9e9f9d551ab4ae34dab5bcdd90564afe" exitCode=0
Oct 30 18:39:25.483628 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:25.483284    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerDied","Data":"9e36175dd7b3eeea81ad19150e4e8d9b9e9f9d551ab4ae34dab5bcdd90564afe"}
Oct 30 18:39:26.491662 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.491619    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"9aded9165615a31461cfcea8397a9e4f6fb8afcb6c6ceb855d7e8d0855d7996d"}
Oct 30 18:39:26.492033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.491692    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"e2702a23c004806149d98533b5f02f22c1c044ee18bf072ddcb92539cefecf3a"}
Oct 30 18:39:26.492033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.491724    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"def05a3f644112a054b61b351b7e8906dee743115b93a0c31ab0c3d2430df94f"}
Oct 30 18:39:26.492033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.491747    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"c073877f97a2eec86ac80c39d8cda274f1e690cec16e6e65b5b138016b4eee67"}
Oct 30 18:39:26.492033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.491782    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"cf969ba0c6bb1e8ed61be71035feb3cd48b65f1bf3b69c7a7cd3b2d6050f90db"}
Oct 30 18:39:26.492033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.491813    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008b7f60-84ef-44fb-a059-66ea1da81563","Type":"ContainerStarted","Data":"ad0037ea23e58b3502602c19792e1e224c2d18d655529498e68b0707ead96627"}
Oct 30 18:39:26.620379 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:26.620330    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.608335623 podStartE2EDuration="3.62031688s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.981257ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:26.620001701 +0000 UTC m=+16.411968998" watchObservedRunningTime="2026-10-30 18:39:26.62031688 +0000 UTC m=+16.412284165"
Oct 30 18:39:27.776683 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.776647    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:27.777101 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.776933    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="prometheus" containerID="cri-o://aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb" gracePeriod=600
Oct 30 18:39:27.777101 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.776972    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="thanos-sidecar" containerID="cri-o://6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c" gracePeriod=600
Oct 30 18:39:27.777101 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.776978    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy" containerID="cri-o://c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25" gracePeriod=600
Oct 30 18:39:27.777101 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.777013    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="config-reloader" containerID="cri-o://92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6" gracePeriod=600
Oct 30 18:39:27.777308 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.776995    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-web" containerID="cri-o://d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1" gracePeriod=600
Oct 30 18:39:27.777308 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:27.777090    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe" gracePeriod=600
Oct 30 18:39:28.034567 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.034515    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.166803 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.166763    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.166949 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.166838    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.166949 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.166884    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.166949 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.166917    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167110 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.166951    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167110 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167005    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167110 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167049    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167110 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167084    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167127    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167167    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167196    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167239    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167450 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167275    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167450 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167309    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167450 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167349    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.167450 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167428    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.169705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167463    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.169705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167503    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.169705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167552    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.169705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.167615    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") pod \"84aa625c-734a-4009-b534-c1acc4c247ee\" (UID: \"84aa625c-734a-4009-b534-c1acc4c247ee\") "
Oct 30 18:39:28.169705 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.168266    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.170698 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.170374    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.171558 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.171529    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.174797 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.173821    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:28.174797 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.174016    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.174797 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.174061    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.175744 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.175708    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.176135 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.176102    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.177651 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.177625    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:28.177819 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.177725    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "kube-api-access-76p4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:28.177915 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.177827    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.178003 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.177960    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.178718 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.178679    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.178864 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.178844    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.178943 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.178916    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.179414 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.179371    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:28.180934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.180862    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:28.180934 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.180914    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.182560 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.182535    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.193156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.193132    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config" pod "84aa625c-734a-4009-b534-c1acc4c247ee" (UID: "84aa625c-734a-4009-b534-c1acc4c247ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:28.268432 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268405    5972 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-config-out\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268432 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268430    5972 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-metrics-client-ca\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268441    5972 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-tls-assets\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268452    5972 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268463    5972 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268472    5972 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268481    5972 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268490    5972 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-db\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268500    5972 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268508    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76p4w\" (UniqueName: \"kubernetes.io/projected/84aa625c-734a-4009-b534-c1acc4c247ee-kube-api-access-76p4w\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268517    5972 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-grpc-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268525    5972 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268534    5972 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268542    5972 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-web-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268551    5972 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-metrics-client-certs\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268559    5972 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268567    5972 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268576    5972 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.268575 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268603    5972 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84aa625c-734a-4009-b534-c1acc4c247ee-secret-kube-rbac-proxy\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.269075 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.268615    5972 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84aa625c-734a-4009-b534-c1acc4c247ee-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504124    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe" exitCode=0
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504157    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25" exitCode=0
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504166    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1" exitCode=0
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504174    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c" exitCode=0
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504182    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6" exitCode=0
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504191    5972 generic.go:356] "Generic (PLEG): container finished" podID="84aa625c-734a-4009-b534-c1acc4c247ee" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb" exitCode=0
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504249    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504320    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504357    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504395    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504432    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504466    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504500    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84aa625c-734a-4009-b534-c1acc4c247ee","Type":"ContainerDied","Data":"eea95d358c5cbe25302945437160ee890a5a9648ce5648630f7b17f8be536d8b"}
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504542    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.505548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.504773    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.514071 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.514053    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.521043 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.521026    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.527648 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.527629    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.534175 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.534160    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.540760 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.540742    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.547556 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.547542    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.553941 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.553926    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.554205 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.554184    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.554253 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.554215    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"} err="failed to get container status \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist"
Oct 30 18:39:28.554253 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.554234    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.554476 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.554459    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.554515 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.554484    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"} err="failed to get container status \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist"
Oct 30 18:39:28.554515 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.554501    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.554763 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.554744    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.554842 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.554767    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"} err="failed to get container status \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist"
Oct 30 18:39:28.554842 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.554783    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.555033 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.555018    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.555068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.555038    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"} err="failed to get container status \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist"
Oct 30 18:39:28.555068 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.555051    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.555257 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.555244    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.555289 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.555261    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"} err="failed to get container status \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist"
Oct 30 18:39:28.555289 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.555273    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.555486 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.555470    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.555518 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.555492    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"} err="failed to get container status \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist"
Oct 30 18:39:28.555518 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.555503    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.556030 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:28.556014    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.556081 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556034    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"} err="failed to get container status \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist"
Oct 30 18:39:28.556081 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556047    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.556293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556273    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"} err="failed to get container status \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist"
Oct 30 18:39:28.556358 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556294    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.556535 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556495    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"} err="failed to get container status \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist"
Oct 30 18:39:28.556535 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556516    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.556784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556745    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"} err="failed to get container status \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist"
Oct 30 18:39:28.556784 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556766    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.556992 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556974    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"} err="failed to get container status \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist"
Oct 30 18:39:28.557032 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.556994    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.557232 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557209    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"} err="failed to get container status \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist"
Oct 30 18:39:28.557281 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557234    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.557468 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557447    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"} err="failed to get container status \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist"
Oct 30 18:39:28.557468 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557465    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.557727 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557708    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"} err="failed to get container status \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist"
Oct 30 18:39:28.557787 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557728    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.557972 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557953    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"} err="failed to get container status \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist"
Oct 30 18:39:28.557972 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.557971    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.558195 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558177    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"} err="failed to get container status \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist"
Oct 30 18:39:28.558252 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558196    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.558403 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558377    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"} err="failed to get container status \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist"
Oct 30 18:39:28.558403 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558403    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.558634 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558572    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"} err="failed to get container status \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist"
Oct 30 18:39:28.558674 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558634    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.558833 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558814    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"} err="failed to get container status \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist"
Oct 30 18:39:28.558901 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.558835    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.559085 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559068    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"} err="failed to get container status \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist"
Oct 30 18:39:28.559129 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559085    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.559324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559305    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"} err="failed to get container status \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist"
Oct 30 18:39:28.559370 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559324    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.559576 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559556    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"} err="failed to get container status \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist"
Oct 30 18:39:28.559635 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559577    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.559820 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559802    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"} err="failed to get container status \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist"
Oct 30 18:39:28.559995 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.559821    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.560042 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560001    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"} err="failed to get container status \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist"
Oct 30 18:39:28.560042 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560019    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.560233 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560215    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"} err="failed to get container status \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist"
Oct 30 18:39:28.560286 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560233    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.560444 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560425    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"} err="failed to get container status \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist"
Oct 30 18:39:28.560516 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560445    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.560679 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560663    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"} err="failed to get container status \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist"
Oct 30 18:39:28.560737 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560680    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.560891 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560873    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"} err="failed to get container status \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist"
Oct 30 18:39:28.560951 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.560892    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.561105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561088    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"} err="failed to get container status \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist"
Oct 30 18:39:28.561146 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561105    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.561336 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561319    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"} err="failed to get container status \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist"
Oct 30 18:39:28.561378 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561337    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.561543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561525    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"} err="failed to get container status \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist"
Oct 30 18:39:28.561599 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561544    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.561736 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561718    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"} err="failed to get container status \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist"
Oct 30 18:39:28.561778 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561736    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.561918 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561901    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"} err="failed to get container status \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist"
Oct 30 18:39:28.561959 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.561919    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.562099 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562078    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"} err="failed to get container status \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist"
Oct 30 18:39:28.562140 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562099    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.562311 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562295    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"} err="failed to get container status \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist"
Oct 30 18:39:28.562355 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562312    5972 scope.go:122] "RemoveContainer" containerID="5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"
Oct 30 18:39:28.562508 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562489    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe"} err="failed to get container status \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": rpc error: code = NotFound desc = could not find container \"5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe\": container with ID starting with 5ca6d8e8fc7988468024226f60a50e5e2b19cf60c69b2f6aa9406659dac83abe not found: ID does not exist"
Oct 30 18:39:28.562545 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562508    5972 scope.go:122] "RemoveContainer" containerID="c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"
Oct 30 18:39:28.562715 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562698    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25"} err="failed to get container status \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": rpc error: code = NotFound desc = could not find container \"c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25\": container with ID starting with c8e0ec92562410bbbcb2c07861866e9b0482d937041026cf8db42fc51d837c25 not found: ID does not exist"
Oct 30 18:39:28.562763 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562715    5972 scope.go:122] "RemoveContainer" containerID="d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"
Oct 30 18:39:28.562906 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562890    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1"} err="failed to get container status \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": rpc error: code = NotFound desc = could not find container \"d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1\": container with ID starting with d3e0a4736960aa5cf92426d6ce59a16a50518fc36d0fbfcd0c5dac06d3e4c4d1 not found: ID does not exist"
Oct 30 18:39:28.562946 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.562905    5972 scope.go:122] "RemoveContainer" containerID="6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"
Oct 30 18:39:28.563101 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563083    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c"} err="failed to get container status \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": rpc error: code = NotFound desc = could not find container \"6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c\": container with ID starting with 6ee037f4f2b515225c6f0572a6b47c74d66907e60739959573db04b006fc2f3c not found: ID does not exist"
Oct 30 18:39:28.563167 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563102    5972 scope.go:122] "RemoveContainer" containerID="92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"
Oct 30 18:39:28.563290 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563276    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6"} err="failed to get container status \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": rpc error: code = NotFound desc = could not find container \"92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6\": container with ID starting with 92a00914dd31368affb92f88af31b26b796b7494b4a3000eb11bf8c1da99f8c6 not found: ID does not exist"
Oct 30 18:39:28.563333 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563291    5972 scope.go:122] "RemoveContainer" containerID="aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"
Oct 30 18:39:28.563479 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563461    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb"} err="failed to get container status \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": rpc error: code = NotFound desc = could not find container \"aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb\": container with ID starting with aac1916206b426c7ae27a32bd2a8c99ad035e32453bf2017a46ed604bfc8dddb not found: ID does not exist"
Oct 30 18:39:28.563522 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563479    5972 scope.go:122] "RemoveContainer" containerID="cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"
Oct 30 18:39:28.563683 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.563666    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550"} err="failed to get container status \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": rpc error: code = NotFound desc = could not find container \"cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550\": container with ID starting with cc62aa513742ef9489db565939d3d524bc8c7920b3d9b0c952e5f5c6aab4c550 not found: ID does not exist"
Oct 30 18:39:28.667103 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.667044    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:28.715098 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.715070    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:28.715875 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.715846    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:28.716273 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716257    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="prometheus"
Oct 30 18:39:28.716332 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716277    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="prometheus"
Oct 30 18:39:28.716332 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716297    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="init-config-reloader"
Oct 30 18:39:28.716332 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716305    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="init-config-reloader"
Oct 30 18:39:28.716332 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716314    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="config-reloader"
Oct 30 18:39:28.716332 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716322    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="config-reloader"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716341    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-web"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716348    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-web"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716368    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716375    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716383    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="thanos-sidecar"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716389    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="thanos-sidecar"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716409    5972 cpu_manager.go:414] "containerMap: removing container" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-thanos"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716417    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-thanos"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716487    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="init-config-reloader"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716496    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="config-reloader"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716504    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-web"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716510    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716515    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="thanos-sidecar"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716522    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="kube-rbac-proxy-thanos"
Oct 30 18:39:28.716552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.716528    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" containerName="prometheus"
Oct 30 18:39:28.722841 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.722814    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775021    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775030    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775056    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775074    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775094    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775097    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 18:39:28.775134 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775105    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 18:39:28.775640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775332    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 18:39:28.775640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775432    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 18:39:28.775640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775462    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 18:39:28.775640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775521    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gg474\""
Oct 30 18:39:28.775640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775525    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 18:39:28.775640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775621    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akuufua83pjtp\""
Oct 30 18:39:28.775948 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.775906    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 18:39:28.824210 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.823898    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 18:39:28.824210 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.824161    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 18:39:28.827936 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.827912    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:28.873400 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873366    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpbv\" (UniqueName: \"kubernetes.io/projected/4e390c06-9942-45ab-b31e-ff7f21ad2993-kube-api-access-5vpbv\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873523 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873411    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873523 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873433    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873523 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873509    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873692 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873539    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-config\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873692 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873564    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873692 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873630    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e390c06-9942-45ab-b31e-ff7f21ad2993-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873837 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873701    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873837 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873759    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873837 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873802    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873940 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873863    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873940 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873890    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873940 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873909    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.873940 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873925    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.874118 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873947    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.874118 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.873962    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e390c06-9942-45ab-b31e-ff7f21ad2993-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.874118 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.874001    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.874118 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.874040    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.874256 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.874136    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.874256 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.874158    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974540 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974508    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974706 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974544    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974706 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974567    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974706 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974615    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-config\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974706 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974650    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974706 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974673    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e390c06-9942-45ab-b31e-ff7f21ad2993-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974708    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974735    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974761    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974788    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974823    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974851    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974874    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974897    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974923    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e390c06-9942-45ab-b31e-ff7f21ad2993-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974944    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.974966 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.974970    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.975498 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.975008    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.975498 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.975033    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.975498 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.975070    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpbv\" (UniqueName: \"kubernetes.io/projected/4e390c06-9942-45ab-b31e-ff7f21ad2993-kube-api-access-5vpbv\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.975498 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.975406    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.977701 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.975942    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.977701 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.976400    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.977701 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.976984    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.977701 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.977311    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.977701 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.977525    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.978500 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.978476    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.979224 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.979161    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e390c06-9942-45ab-b31e-ff7f21ad2993-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.979427 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.979402    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.979965 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.979918    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.979965 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.979924    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.980207 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.980182    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.980312 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.980181    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.980312 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.980266    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e390c06-9942-45ab-b31e-ff7f21ad2993-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.980522 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.980496    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.980646 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.980635    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e390c06-9942-45ab-b31e-ff7f21ad2993-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.980828 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.980812    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-config\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.981192 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.981175    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:28.981567 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:28.981551    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e390c06-9942-45ab-b31e-ff7f21ad2993-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:29.034341 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:29.034314    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpbv\" (UniqueName: \"kubernetes.io/projected/4e390c06-9942-45ab-b31e-ff7f21ad2993-kube-api-access-5vpbv\") pod \"prometheus-k8s-0\" (UID: \"4e390c06-9942-45ab-b31e-ff7f21ad2993\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:29.055030 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:29.055009    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:29.485874 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:29.485845    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 18:39:29.488993 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:39:29.488969    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e390c06_9942_45ab_b31e_ff7f21ad2993.slice/crio-c15e4dc1c9967ec434d7a098a7b4e74577c961cd0175ae993cf8297aa59044c4 WatchSource:0}: Error finding container c15e4dc1c9967ec434d7a098a7b4e74577c961cd0175ae993cf8297aa59044c4: Status 404 returned error can't find the container with id c15e4dc1c9967ec434d7a098a7b4e74577c961cd0175ae993cf8297aa59044c4
Oct 30 18:39:29.509323 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:29.509299    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"c15e4dc1c9967ec434d7a098a7b4e74577c961cd0175ae993cf8297aa59044c4"}
Oct 30 18:39:30.340108 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:30.340074    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84aa625c-734a-4009-b534-c1acc4c247ee" path="/var/lib/kubelet/pods/84aa625c-734a-4009-b534-c1acc4c247ee/volumes"
Oct 30 18:39:30.515492 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:30.515460    5972 generic.go:356] "Generic (PLEG): container finished" podID="4e390c06-9942-45ab-b31e-ff7f21ad2993" containerID="3befee5259ff28fb08855055c4b5b6748484678493efa2748a5bf3e997b84cc5" exitCode=0
Oct 30 18:39:30.515716 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:30.515519    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerDied","Data":"3befee5259ff28fb08855055c4b5b6748484678493efa2748a5bf3e997b84cc5"}
Oct 30 18:39:31.522798 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.522760    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"b0b87a568d71e2814f351dc713b00535ba92c656c447da8d7b2967489baaecdb"}
Oct 30 18:39:31.523156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.522879    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"8b8b1a2a3c69f100a4a81ccfa5b1ae44d1fdb8d4add49c871e45288df4781a3f"}
Oct 30 18:39:31.523156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.522925    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"dd130c6a53834daf530086823e38c733066e4cd5d8969a2a52386c2451fc4065"}
Oct 30 18:39:31.523156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.522961    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"338414d3f83b1006ae7f3124ce22dcb4fb5e91e0648e3782fed9d888d3e678f2"}
Oct 30 18:39:31.523156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.523003    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"c81b43992db5caccc2c0ded4c94af3bf1b96209e8dccae861d56d71219972ebf"}
Oct 30 18:39:31.523156 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.523039    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e390c06-9942-45ab-b31e-ff7f21ad2993","Type":"ContainerStarted","Data":"c8022eb6b544e9cf6df0e72a72465b7dae402716850d5ab8195c3f81d31eb829"}
Oct 30 18:39:31.651995 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:31.651940    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.634685539 podStartE2EDuration="3.651924209s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.23867ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:31.651492535 +0000 UTC m=+21.443459826" watchObservedRunningTime="2026-10-30 18:39:31.651924209 +0000 UTC m=+21.443891502"
Oct 30 18:39:34.055180 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:34.055137    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:34.477137 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:34.477114    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fdf4b"
Oct 30 18:39:42.176367 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:42.176338    5972 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:42.180130 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:42.180104    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-58f458b77f-264xl"
Oct 30 18:39:47.691351 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:47.691310    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mdb7t"
Oct 30 18:39:48.993127 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:48.993097    5972 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:49.009263 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:49.009241    5972 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:49.071385 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:39:49.071360    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 18:39:52.871656 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:39:52.871616    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:40:22.874893 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:40:22.872346    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:40:52.870135 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:40:52.870057    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:41:22.869270 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:41:22.869233    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:41:52.869501 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:41:52.869467    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:42:22.869812 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:42:22.869733    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:42:52.870636 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:42:52.870595    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:43:22.869236 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:43:22.869199    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:43:52.874336 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:43:52.874262    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:44:10.377822 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:44:10.377787    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:44:10.382251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:44:10.382230    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:44:22.869828 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:44:22.869786    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:44:52.870529 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:44:52.870487    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:45:22.870609 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:45:22.870506    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:45:52.869240 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:45:52.869202    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:46:22.869741 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:46:22.869706    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:46:52.869046 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:46:52.868962    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:47:00.779734 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.779704    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kg4zb"]
Oct 30 18:47:00.782950 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.782933    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kg4zb"
Oct 30 18:47:00.824672 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.824648    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache\") pod \"s3-init-kg4zb\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") " pod="kserve/s3-init-kg4zb"
Oct 30 18:47:00.824800 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.824716    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4b4t\" (UniqueName: \"kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t\") pod \"s3-init-kg4zb\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") " pod="kserve/s3-init-kg4zb"
Oct 30 18:47:00.837190 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.837166    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kg4zb"]
Oct 30 18:47:00.881434 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.881405    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 18:47:00.881544 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.881443    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 18:47:00.881544 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.881447    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:47:00.881544 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.881476    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 18:47:00.881679 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.881570    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:47:00.925445 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.925426    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4b4t\" (UniqueName: \"kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t\") pod \"s3-init-kg4zb\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") " pod="kserve/s3-init-kg4zb"
Oct 30 18:47:00.925552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.925474    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache\") pod \"s3-init-kg4zb\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") " pod="kserve/s3-init-kg4zb"
Oct 30 18:47:00.925816 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.925802    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache\") pod \"s3-init-kg4zb\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") " pod="kserve/s3-init-kg4zb"
Oct 30 18:47:00.985033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:00.985004    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4b4t\" (UniqueName: \"kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t\") pod \"s3-init-kg4zb\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") " pod="kserve/s3-init-kg4zb"
Oct 30 18:47:01.108893 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:01.108839    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kg4zb"
Oct 30 18:47:01.471902 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:01.471877    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kg4zb"]
Oct 30 18:47:01.474771 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:47:01.474746    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod135a46df_6019_437d_9bf0_d2618a076345.slice/crio-984422769e0bf67eb3b1c74a924ba9ed3678dd4c459f751d9979482234bc378b WatchSource:0}: Error finding container 984422769e0bf67eb3b1c74a924ba9ed3678dd4c459f751d9979482234bc378b: Status 404 returned error can't find the container with id 984422769e0bf67eb3b1c74a924ba9ed3678dd4c459f751d9979482234bc378b
Oct 30 18:47:01.476875 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:01.476859    5972 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:47:02.165517 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:02.165480    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kg4zb" event={"ID":"135a46df-6019-437d-9bf0-d2618a076345","Type":"ContainerStarted","Data":"984422769e0bf67eb3b1c74a924ba9ed3678dd4c459f751d9979482234bc378b"}
Oct 30 18:47:10.760523 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:10.760498    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 18:47:11.199077 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:11.199046    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kg4zb" event={"ID":"135a46df-6019-437d-9bf0-d2618a076345","Type":"ContainerStarted","Data":"a8fa18e3887951091083f45b605481501e0dc73bc1a8d24f8f6edfac21da60db"}
Oct 30 18:47:13.209096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:13.209064    5972 generic.go:356] "Generic (PLEG): container finished" podID="135a46df-6019-437d-9bf0-d2618a076345" containerID="a8fa18e3887951091083f45b605481501e0dc73bc1a8d24f8f6edfac21da60db" exitCode=0
Oct 30 18:47:13.209434 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:13.209132    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kg4zb" event={"ID":"135a46df-6019-437d-9bf0-d2618a076345","Type":"ContainerDied","Data":"a8fa18e3887951091083f45b605481501e0dc73bc1a8d24f8f6edfac21da60db"}
Oct 30 18:47:18.210064 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:18.210040    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 18:47:19.236536 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:19.236497    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kg4zb" event={"ID":"135a46df-6019-437d-9bf0-d2618a076345","Type":"ContainerStarted","Data":"b5e17409b540d465bc03c23811935dc9f669e7c61defc00663c7c382d0f63932"}
Oct 30 18:47:19.349448 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:47:19.349393    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-kg4zb" podStartSLOduration=3.173540814 podStartE2EDuration="19.349378421s" totalImagesPullingTime="14.177798471s" totalInitContainerRuntime="1.998039136s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:19.348733691 +0000 UTC m=+489.140701002" watchObservedRunningTime="2026-10-30 18:47:19.349378421 +0000 UTC m=+489.141345699"
Oct 30 18:47:22.869443 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:47:22.869406    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:47:52.870220 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:47:52.870183    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:48:22.869498 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:48:22.869367    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:48:23.468400 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:23.468370    5972 generic.go:356] "Generic (PLEG): container finished" podID="135a46df-6019-437d-9bf0-d2618a076345" containerID="b5e17409b540d465bc03c23811935dc9f669e7c61defc00663c7c382d0f63932" exitCode=0
Oct 30 18:48:23.468557 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:23.468429    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kg4zb" event={"ID":"135a46df-6019-437d-9bf0-d2618a076345","Type":"ContainerDied","Data":"b5e17409b540d465bc03c23811935dc9f669e7c61defc00663c7c382d0f63932"}
Oct 30 18:48:24.595138 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.595114    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kg4zb"
Oct 30 18:48:24.622150 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.622129    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t\" (UniqueName: \"kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t\") pod \"135a46df-6019-437d-9bf0-d2618a076345\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") "
Oct 30 18:48:24.622282 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.622234    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache\") pod \"135a46df-6019-437d-9bf0-d2618a076345\" (UID: \"135a46df-6019-437d-9bf0-d2618a076345\") "
Oct 30 18:48:24.624341 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.624314    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t" pod "135a46df-6019-437d-9bf0-d2618a076345" (UID: "135a46df-6019-437d-9bf0-d2618a076345"). InnerVolumeSpecName "kube-api-access-t4b4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:24.640175 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.640145    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache" pod "135a46df-6019-437d-9bf0-d2618a076345" (UID: "135a46df-6019-437d-9bf0-d2618a076345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:24.723424 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.723357    5972 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135a46df-6019-437d-9bf0-d2618a076345-model-cache\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:24.723424 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:24.723380    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4b4t\" (UniqueName: \"kubernetes.io/projected/135a46df-6019-437d-9bf0-d2618a076345-kube-api-access-t4b4t\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:25.476467 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:25.476441    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kg4zb"
Oct 30 18:48:25.476651 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:25.476441    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kg4zb" event={"ID":"135a46df-6019-437d-9bf0-d2618a076345","Type":"ContainerDied","Data":"984422769e0bf67eb3b1c74a924ba9ed3678dd4c459f751d9979482234bc378b"}
Oct 30 18:48:25.476651 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:48:25.476547    5972 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="984422769e0bf67eb3b1c74a924ba9ed3678dd4c459f751d9979482234bc378b"
Oct 30 18:48:52.871516 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:48:52.871482    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:49:01.429140 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.429110    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6srx5"]
Oct 30 18:49:01.429568 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.429438    5972 cpu_manager.go:414] "containerMap: removing container" podUID="135a46df-6019-437d-9bf0-d2618a076345" containerName="s3-init"
Oct 30 18:49:01.429568 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.429450    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135a46df-6019-437d-9bf0-d2618a076345" containerName="s3-init"
Oct 30 18:49:01.429568 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.429475    5972 cpu_manager.go:414] "containerMap: removing container" podUID="135a46df-6019-437d-9bf0-d2618a076345" containerName="download-hf-model"
Oct 30 18:49:01.429568 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.429480    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135a46df-6019-437d-9bf0-d2618a076345" containerName="download-hf-model"
Oct 30 18:49:01.429568 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.429535    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="135a46df-6019-437d-9bf0-d2618a076345" containerName="s3-init"
Oct 30 18:49:01.432666 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.432626    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.485380 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.485332    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 18:49:01.485380 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.485364    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:49:01.529413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.529382    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlxnx\" (UniqueName: \"kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx\") pod \"s3-tls-init-custom-6srx5\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") " pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.529540 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.529430    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache\") pod \"s3-tls-init-custom-6srx5\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") " pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.533096 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.533077    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Oct 30 18:49:01.533163 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.533142    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 18:49:01.533163 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.533155    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:49:01.533805 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.533790    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6srx5"]
Oct 30 18:49:01.630382 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.630356    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache\") pod \"s3-tls-init-custom-6srx5\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") " pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.630505 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.630414    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlxnx\" (UniqueName: \"kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx\") pod \"s3-tls-init-custom-6srx5\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") " pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.630734 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.630717    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache\") pod \"s3-tls-init-custom-6srx5\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") " pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.689847 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.689826    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlxnx\" (UniqueName: \"kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx\") pod \"s3-tls-init-custom-6srx5\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") " pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:01.764957 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:01.764926    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:02.128490 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:02.128412    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6srx5"]
Oct 30 18:49:02.132025 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:49:02.131991    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod945f7045_e38a_42be_a05d_9ee922814031.slice/crio-a06914880d0e086f7237de06c4eb7c9eab21efcc83e03c8d2f7cbec39579b9cf WatchSource:0}: Error finding container a06914880d0e086f7237de06c4eb7c9eab21efcc83e03c8d2f7cbec39579b9cf: Status 404 returned error can't find the container with id a06914880d0e086f7237de06c4eb7c9eab21efcc83e03c8d2f7cbec39579b9cf
Oct 30 18:49:02.615515 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:02.613715    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6srx5" event={"ID":"945f7045-e38a-42be-a05d-9ee922814031","Type":"ContainerStarted","Data":"0efe7cd876fd97c0e0cac46c7e81eef1ebd5df167ee7ea3c3088a731959af733"}
Oct 30 18:49:02.615515 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:02.613782    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6srx5" event={"ID":"945f7045-e38a-42be-a05d-9ee922814031","Type":"ContainerStarted","Data":"a06914880d0e086f7237de06c4eb7c9eab21efcc83e03c8d2f7cbec39579b9cf"}
Oct 30 18:49:04.625827 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:04.625792    5972 generic.go:356] "Generic (PLEG): container finished" podID="945f7045-e38a-42be-a05d-9ee922814031" containerID="0efe7cd876fd97c0e0cac46c7e81eef1ebd5df167ee7ea3c3088a731959af733" exitCode=0
Oct 30 18:49:04.626256 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:04.625866    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6srx5" event={"ID":"945f7045-e38a-42be-a05d-9ee922814031","Type":"ContainerDied","Data":"0efe7cd876fd97c0e0cac46c7e81eef1ebd5df167ee7ea3c3088a731959af733"}
Oct 30 18:49:05.631757 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:05.631723    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6srx5" event={"ID":"945f7045-e38a-42be-a05d-9ee922814031","Type":"ContainerStarted","Data":"6b172a21f8f48004888e70996f147b805fe12c90e15a3b38935042ad58fd8999"}
Oct 30 18:49:05.746065 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:05.746018    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6srx5" podStartSLOduration=3.189081314 podStartE2EDuration="4.74600374s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.556922426s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:05.745369588 +0000 UTC m=+595.537336877" watchObservedRunningTime="2026-10-30 18:49:05.74600374 +0000 UTC m=+595.537971027"
Oct 30 18:49:10.416482 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:10.416459    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:49:10.418749 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:10.418727    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:49:11.655963 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:11.655929    5972 generic.go:356] "Generic (PLEG): container finished" podID="945f7045-e38a-42be-a05d-9ee922814031" containerID="6b172a21f8f48004888e70996f147b805fe12c90e15a3b38935042ad58fd8999" exitCode=0
Oct 30 18:49:11.656309 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:11.655973    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6srx5" event={"ID":"945f7045-e38a-42be-a05d-9ee922814031","Type":"ContainerDied","Data":"6b172a21f8f48004888e70996f147b805fe12c90e15a3b38935042ad58fd8999"}
Oct 30 18:49:12.784201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.784177    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:12.814492 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.814466    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache\") pod \"945f7045-e38a-42be-a05d-9ee922814031\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") "
Oct 30 18:49:12.814619 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.814498    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx\" (UniqueName: \"kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx\") pod \"945f7045-e38a-42be-a05d-9ee922814031\" (UID: \"945f7045-e38a-42be-a05d-9ee922814031\") "
Oct 30 18:49:12.816555 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.816529    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx" pod "945f7045-e38a-42be-a05d-9ee922814031" (UID: "945f7045-e38a-42be-a05d-9ee922814031"). InnerVolumeSpecName "kube-api-access-jlxnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:12.837430 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.837405    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache" pod "945f7045-e38a-42be-a05d-9ee922814031" (UID: "945f7045-e38a-42be-a05d-9ee922814031"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:49:12.915435 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.915378    5972 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f7045-e38a-42be-a05d-9ee922814031-model-cache\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:12.915435 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:12.915414    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlxnx\" (UniqueName: \"kubernetes.io/projected/945f7045-e38a-42be-a05d-9ee922814031-kube-api-access-jlxnx\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:13.665083 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:13.665044    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6srx5" event={"ID":"945f7045-e38a-42be-a05d-9ee922814031","Type":"ContainerDied","Data":"a06914880d0e086f7237de06c4eb7c9eab21efcc83e03c8d2f7cbec39579b9cf"}
Oct 30 18:49:13.665246 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:13.665094    5972 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a06914880d0e086f7237de06c4eb7c9eab21efcc83e03c8d2f7cbec39579b9cf"
Oct 30 18:49:13.665246 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:13.665057    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6srx5"
Oct 30 18:49:17.860818 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.860786    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-9lw8s"]
Oct 30 18:49:17.861191 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.861112    5972 cpu_manager.go:414] "containerMap: removing container" podUID="945f7045-e38a-42be-a05d-9ee922814031" containerName="download-hf-model"
Oct 30 18:49:17.861191 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.861122    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945f7045-e38a-42be-a05d-9ee922814031" containerName="download-hf-model"
Oct 30 18:49:17.861191 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.861132    5972 cpu_manager.go:414] "containerMap: removing container" podUID="945f7045-e38a-42be-a05d-9ee922814031" containerName="s3-tls-init-custom"
Oct 30 18:49:17.861191 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.861137    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945f7045-e38a-42be-a05d-9ee922814031" containerName="s3-tls-init-custom"
Oct 30 18:49:17.861318 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.861203    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="945f7045-e38a-42be-a05d-9ee922814031" containerName="s3-tls-init-custom"
Oct 30 18:49:17.864770 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.864754    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:17.918276 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.918255    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9lw8s"]
Oct 30 18:49:17.955009 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.954988    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6htj\" (UniqueName: \"kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj\") pod \"s3-tls-init-serving-9lw8s\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") " pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:17.955132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.955036    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache\") pod \"s3-tls-init-serving-9lw8s\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") " pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:17.965415 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.965375    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 18:49:17.965415 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.965383    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:49:17.965621 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.965439    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 30 18:49:17.965621 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.965445    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:49:17.965621 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:17.965444    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 18:49:18.055711 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.055682    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6htj\" (UniqueName: \"kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj\") pod \"s3-tls-init-serving-9lw8s\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") " pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:18.055833 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.055733    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache\") pod \"s3-tls-init-serving-9lw8s\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") " pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:18.056058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.056042    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache\") pod \"s3-tls-init-serving-9lw8s\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") " pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:18.115196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.115144    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6htj\" (UniqueName: \"kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj\") pod \"s3-tls-init-serving-9lw8s\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") " pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:18.191217 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.191187    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:18.555235 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.555207    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9lw8s"]
Oct 30 18:49:18.558920 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:49:18.558897    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcbe0fde_bff4_4c66_93a1_13653ad5d211.slice/crio-cb630640907513ffa333f45b8568f651b6d452ffa484de92e04b21b4ec2c988c WatchSource:0}: Error finding container cb630640907513ffa333f45b8568f651b6d452ffa484de92e04b21b4ec2c988c: Status 404 returned error can't find the container with id cb630640907513ffa333f45b8568f651b6d452ffa484de92e04b21b4ec2c988c
Oct 30 18:49:18.685402 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.685372    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9lw8s" event={"ID":"fcbe0fde-bff4-4c66-93a1-13653ad5d211","Type":"ContainerStarted","Data":"97a69e8a6b73d79ebd60d9df8333b8f55ed57c59a0d1aab71a46f9dbdb725239"}
Oct 30 18:49:18.685529 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:18.685412    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9lw8s" event={"ID":"fcbe0fde-bff4-4c66-93a1-13653ad5d211","Type":"ContainerStarted","Data":"cb630640907513ffa333f45b8568f651b6d452ffa484de92e04b21b4ec2c988c"}
Oct 30 18:49:20.693064 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:20.693024    5972 generic.go:356] "Generic (PLEG): container finished" podID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerID="97a69e8a6b73d79ebd60d9df8333b8f55ed57c59a0d1aab71a46f9dbdb725239" exitCode=0
Oct 30 18:49:20.693437 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:20.693099    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9lw8s" event={"ID":"fcbe0fde-bff4-4c66-93a1-13653ad5d211","Type":"ContainerDied","Data":"97a69e8a6b73d79ebd60d9df8333b8f55ed57c59a0d1aab71a46f9dbdb725239"}
Oct 30 18:49:21.698293 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:21.698253    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9lw8s" event={"ID":"fcbe0fde-bff4-4c66-93a1-13653ad5d211","Type":"ContainerStarted","Data":"9e701ce03384b19892c9ba778d425273d1b5ebb546caa2ff94098918f52edd2b"}
Oct 30 18:49:21.810890 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:21.810843    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-9lw8s" podStartSLOduration=3.301231697 podStartE2EDuration="4.810830438s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.509598741s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:21.810623905 +0000 UTC m=+611.602591192" watchObservedRunningTime="2026-10-30 18:49:21.810830438 +0000 UTC m=+611.602797723"
Oct 30 18:49:22.870117 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:49:22.870082    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:49:26.718678 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:26.718653    5972 generic.go:356] "Generic (PLEG): container finished" podID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerID="9e701ce03384b19892c9ba778d425273d1b5ebb546caa2ff94098918f52edd2b" exitCode=0
Oct 30 18:49:26.719079 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:26.718731    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9lw8s" event={"ID":"fcbe0fde-bff4-4c66-93a1-13653ad5d211","Type":"ContainerDied","Data":"9e701ce03384b19892c9ba778d425273d1b5ebb546caa2ff94098918f52edd2b"}
Oct 30 18:49:27.847893 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:27.847871    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:27.938473 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:27.938453    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj\" (UniqueName: \"kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj\") pod \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") "
Oct 30 18:49:27.938598 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:27.938483    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache\") pod \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\" (UID: \"fcbe0fde-bff4-4c66-93a1-13653ad5d211\") "
Oct 30 18:49:27.940543 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:27.940518    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj" pod "fcbe0fde-bff4-4c66-93a1-13653ad5d211" (UID: "fcbe0fde-bff4-4c66-93a1-13653ad5d211"). InnerVolumeSpecName "kube-api-access-d6htj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:27.965101 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:27.965076    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache" pod "fcbe0fde-bff4-4c66-93a1-13653ad5d211" (UID: "fcbe0fde-bff4-4c66-93a1-13653ad5d211"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:49:28.039348 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:28.039283    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6htj\" (UniqueName: \"kubernetes.io/projected/fcbe0fde-bff4-4c66-93a1-13653ad5d211-kube-api-access-d6htj\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:28.039348 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:28.039308    5972 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbe0fde-bff4-4c66-93a1-13653ad5d211-model-cache\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:28.728248 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:28.728214    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9lw8s" event={"ID":"fcbe0fde-bff4-4c66-93a1-13653ad5d211","Type":"ContainerDied","Data":"cb630640907513ffa333f45b8568f651b6d452ffa484de92e04b21b4ec2c988c"}
Oct 30 18:49:28.728408 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:28.728256    5972 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb630640907513ffa333f45b8568f651b6d452ffa484de92e04b21b4ec2c988c"
Oct 30 18:49:28.728408 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:28.728226    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9lw8s"
Oct 30 18:49:50.001279 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.001195    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"]
Oct 30 18:49:50.001718 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.001531    5972 cpu_manager.go:414] "containerMap: removing container" podUID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerName="s3-tls-init-serving"
Oct 30 18:49:50.001718 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.001542    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerName="s3-tls-init-serving"
Oct 30 18:49:50.001718 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.001566    5972 cpu_manager.go:414] "containerMap: removing container" podUID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerName="download-hf-model"
Oct 30 18:49:50.001718 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.001571    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerName="download-hf-model"
Oct 30 18:49:50.001718 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.001649    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbe0fde-bff4-4c66-93a1-13653ad5d211" containerName="s3-tls-init-serving"
Oct 30 18:49:50.006201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.006178    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.057482 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.057451    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 18:49:50.105055 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.105022    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:49:50.105055 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.105043    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 18:49:50.105055 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.105022    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 18:49:50.105423 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.105095    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Oct 30 18:49:50.105423 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.105112    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Oct 30 18:49:50.105423 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.105165    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 18:49:50.106348 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.106323    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"]
Oct 30 18:49:50.134812 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.134775    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.135017 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.134843    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.135017 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.134966    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.135127 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.135047    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfjsg\" (UniqueName: \"kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.135127 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.135112    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236043 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236006    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236071    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236131    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236193    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfjsg\" (UniqueName: \"kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236251 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236229    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236728 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236697    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236882 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236864    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.236928 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.236902    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.238781 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.238760    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.296185 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.296101    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfjsg\" (UniqueName: \"kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg\") pod \"isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.326139 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.326099    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:49:50.747547 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.747405    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"]
Oct 30 18:49:50.750769 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:49:50.750731    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3e514f_41ba_4792_8064_300bdb6cad13.slice/crio-6843d9c121c6fdc84e7fe08396a96f19a26a191a54ee58d02764caebd1402c2f WatchSource:0}: Error finding container 6843d9c121c6fdc84e7fe08396a96f19a26a191a54ee58d02764caebd1402c2f: Status 404 returned error can't find the container with id 6843d9c121c6fdc84e7fe08396a96f19a26a191a54ee58d02764caebd1402c2f
Oct 30 18:49:50.810032 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:50.809967    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerStarted","Data":"6843d9c121c6fdc84e7fe08396a96f19a26a191a54ee58d02764caebd1402c2f"}
Oct 30 18:49:51.815368 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:51.815333    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerStarted","Data":"fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4"}
Oct 30 18:49:52.871607 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:49:52.871556    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:49:54.829239 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:54.829201    5972 generic.go:356] "Generic (PLEG): container finished" podID="da3e514f-41ba-4792-8064-300bdb6cad13" containerID="fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4" exitCode=0
Oct 30 18:49:54.829763 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:49:54.829280    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerDied","Data":"fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4"}
Oct 30 18:50:15.928098 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:15.928059    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerStarted","Data":"d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0"}
Oct 30 18:50:22.876299 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:50:22.876254    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:50:24.976218 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:24.976180    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerStarted","Data":"2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7"}
Oct 30 18:50:34.013419 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:34.013385    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerStarted","Data":"966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a"}
Oct 30 18:50:34.013904 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:34.013622    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:50:34.013904 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:34.013693    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:50:34.015245 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:34.015196    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:50:34.019219 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:34.019191    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:50:34.229039 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:34.228987    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podStartSLOduration=3.147486201 podStartE2EDuration="45.228969633s" totalImagesPullingTime="38.210362641s" totalInitContainerRuntime="3.871120791s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:34.180470343 +0000 UTC m=+683.972437656" watchObservedRunningTime="2026-10-30 18:50:34.228969633 +0000 UTC m=+684.020936922"
Oct 30 18:50:35.018317 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:35.018274    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:50:35.018947 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:35.018472    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:50:35.019482 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:35.019455    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:36.022008 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:36.021965    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:50:36.022423 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:36.022253    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:46.021946 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:46.021895    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:50:46.022713 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:46.022365    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:52.869886 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:50:52.869844    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:50:56.022329 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:56.022284    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:50:56.022764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:50:56.022738    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:06.022322 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:06.022274    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:51:06.022753 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:06.022684    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:16.022094 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:16.022049    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:51:16.022507 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:16.022462    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:22.869748 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:51:22.869658    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:51:26.022182 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:26.022138    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:51:26.022756 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:26.022685    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:36.023165 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:36.023131    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:51:36.023648 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:36.023424    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:51:45.437206 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:45.437175    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"]
Oct 30 18:51:45.437607 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:45.437428    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" containerID="cri-o://d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0" gracePeriod=30
Oct 30 18:51:45.437607 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:45.437433    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" containerID="cri-o://966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a" gracePeriod=30
Oct 30 18:51:45.437607 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:45.437477    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" containerID="cri-o://2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7" gracePeriod=30
Oct 30 18:51:46.021835 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:46.021789    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:51:46.022130 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:46.022105    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:46.296786 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:46.296696    5972 generic.go:356] "Generic (PLEG): container finished" podID="da3e514f-41ba-4792-8064-300bdb6cad13" containerID="2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7" exitCode=0
Oct 30 18:51:46.296786 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:46.296766    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerDied","Data":"2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7"}
Oct 30 18:51:49.014548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:49.014505    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Oct 30 18:51:50.315727 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:50.315645    5972 generic.go:356] "Generic (PLEG): container finished" podID="da3e514f-41ba-4792-8064-300bdb6cad13" containerID="d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0" exitCode=0
Oct 30 18:51:50.316086 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:50.315718    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerDied","Data":"d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0"}
Oct 30 18:51:52.870087 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:51:52.870052    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:51:54.014431 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:54.014390    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Oct 30 18:51:56.021831 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:56.021782    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:51:56.022234 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:56.022050    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:58.586404 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.586366    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"]
Oct 30 18:51:58.590290 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.590268    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.641315 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.641294    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:51:58.689170 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.689146    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:51:58.689358 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.689191    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-primary-27eef3-kube-rbac-proxy-sar-config\""
Oct 30 18:51:58.689433 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.689191    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-primary-27eef3-predictor-serving-cert\""
Oct 30 18:51:58.689433 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.689210    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:51:58.689433 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.689219    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:51:58.689433 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.689255    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:51:58.690227 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.690210    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"]
Oct 30 18:51:58.757824 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.757796    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.757956 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.757834    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwjg\" (UniqueName: \"kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.757956 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.757867    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.758044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.757976    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.758044 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.758009    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.859525 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.859463    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.859525 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.859501    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwjg\" (UniqueName: \"kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.859722 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.859565    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.859722 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.859617    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.859722 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.859643    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.859991 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.859969    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.860208 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.860191    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.860249 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.860212    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.862154 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.862134    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:58.919173 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:58.919151    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwjg\" (UniqueName: \"kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg\") pod \"isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") " pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:59.014181 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:59.014153    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Oct 30 18:51:59.014298 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:59.014269    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:51:59.207230 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:59.207199    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:51:59.628354 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:51:59.628125    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"]
Oct 30 18:51:59.631223 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:51:59.631191    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17d4300_f366_458c_b8c7_596546159e3a.slice/crio-b2a4dbd38a4fa717df1a2e0a47b6fc43b40db691cbaa3c07a60bcd379eab89b2 WatchSource:0}: Error finding container b2a4dbd38a4fa717df1a2e0a47b6fc43b40db691cbaa3c07a60bcd379eab89b2: Status 404 returned error can't find the container with id b2a4dbd38a4fa717df1a2e0a47b6fc43b40db691cbaa3c07a60bcd379eab89b2
Oct 30 18:52:00.354845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:00.354817    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerStarted","Data":"05f90cc1dadee10d7b09dee19d48a509706307b6c72ea5eee29524e27bae9f98"}
Oct 30 18:52:00.355022 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:00.354868    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerStarted","Data":"b2a4dbd38a4fa717df1a2e0a47b6fc43b40db691cbaa3c07a60bcd379eab89b2"}
Oct 30 18:52:04.014065 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:04.014025    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Oct 30 18:52:04.370524 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:04.370444    5972 generic.go:356] "Generic (PLEG): container finished" podID="a17d4300-f366-458c-b8c7-596546159e3a" containerID="05f90cc1dadee10d7b09dee19d48a509706307b6c72ea5eee29524e27bae9f98" exitCode=0
Oct 30 18:52:04.370524 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:04.370513    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerDied","Data":"05f90cc1dadee10d7b09dee19d48a509706307b6c72ea5eee29524e27bae9f98"}
Oct 30 18:52:05.376677 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:05.376646    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerStarted","Data":"a3707578c61882cf092512f3fc98a4a8b16b0d5fb880ac47b178a7968b4f5cfa"}
Oct 30 18:52:05.377065 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:05.376707    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerStarted","Data":"3cc161125c9a4ff61b406f829b661af5f6d3b6736b63dcb8d05627246878722c"}
Oct 30 18:52:05.377065 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:05.377050    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:52:05.377196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:05.377181    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:52:05.378490 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:05.378458    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:05.496597 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:05.494331    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podStartSLOduration=3.68151738 podStartE2EDuration="7.494318394s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812801014s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:05.49344492 +0000 UTC m=+775.285412206" watchObservedRunningTime="2026-10-30 18:52:05.494318394 +0000 UTC m=+775.286285683"
Oct 30 18:52:06.021901 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:06.021852    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Oct 30 18:52:06.022104 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:06.022067    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:52:06.022176 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:06.022135    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:06.022231 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:06.022203    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:52:06.380813 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:06.380712    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:09.014552 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:09.014515    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Oct 30 18:52:11.385106 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:11.385082    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:52:11.385692 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:11.385670    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:14.013972 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:14.013924    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Oct 30 18:52:15.578545 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.578522    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:52:15.594653 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.594631    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location\") pod \"da3e514f-41ba-4792-8064-300bdb6cad13\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") "
Oct 30 18:52:15.594758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.594670    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert\") pod \"da3e514f-41ba-4792-8064-300bdb6cad13\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") "
Oct 30 18:52:15.594758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.594705    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config\") pod \"da3e514f-41ba-4792-8064-300bdb6cad13\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") "
Oct 30 18:52:15.594758 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.594733    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls\") pod \"da3e514f-41ba-4792-8064-300bdb6cad13\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") "
Oct 30 18:52:15.594920 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.594795    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg\" (UniqueName: \"kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg\") pod \"da3e514f-41ba-4792-8064-300bdb6cad13\" (UID: \"da3e514f-41ba-4792-8064-300bdb6cad13\") "
Oct 30 18:52:15.595032 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.594941    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location" pod "da3e514f-41ba-4792-8064-300bdb6cad13" (UID: "da3e514f-41ba-4792-8064-300bdb6cad13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:15.595130 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.595032    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert" pod "da3e514f-41ba-4792-8064-300bdb6cad13" (UID: "da3e514f-41ba-4792-8064-300bdb6cad13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:15.595130 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.595074    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config" pod "da3e514f-41ba-4792-8064-300bdb6cad13" (UID: "da3e514f-41ba-4792-8064-300bdb6cad13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:15.597199 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.597171    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls" pod "da3e514f-41ba-4792-8064-300bdb6cad13" (UID: "da3e514f-41ba-4792-8064-300bdb6cad13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:15.597260 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.597192    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg" pod "da3e514f-41ba-4792-8064-300bdb6cad13" (UID: "da3e514f-41ba-4792-8064-300bdb6cad13"). InnerVolumeSpecName "kube-api-access-rfjsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:15.695558 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.695480    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:15.695558 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.695506    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3e514f-41ba-4792-8064-300bdb6cad13-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:15.695558 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.695517    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfjsg\" (UniqueName: \"kubernetes.io/projected/da3e514f-41ba-4792-8064-300bdb6cad13-kube-api-access-rfjsg\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:15.695558 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.695526    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da3e514f-41ba-4792-8064-300bdb6cad13-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:15.695558 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:15.695535    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da3e514f-41ba-4792-8064-300bdb6cad13-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:16.420247 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.420214    5972 generic.go:356] "Generic (PLEG): container finished" podID="da3e514f-41ba-4792-8064-300bdb6cad13" containerID="966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a" exitCode=0
Oct 30 18:52:16.420426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.420275    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerDied","Data":"966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a"}
Oct 30 18:52:16.420426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.420279    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"
Oct 30 18:52:16.420426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.420328    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g" event={"ID":"da3e514f-41ba-4792-8064-300bdb6cad13","Type":"ContainerDied","Data":"6843d9c121c6fdc84e7fe08396a96f19a26a191a54ee58d02764caebd1402c2f"}
Oct 30 18:52:16.420426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.420357    5972 scope.go:122] "RemoveContainer" containerID="966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a"
Oct 30 18:52:16.428752 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.428734    5972 scope.go:122] "RemoveContainer" containerID="2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7"
Oct 30 18:52:16.436001 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.435974    5972 scope.go:122] "RemoveContainer" containerID="d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0"
Oct 30 18:52:16.443233 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.443070    5972 scope.go:122] "RemoveContainer" containerID="fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4"
Oct 30 18:52:16.450133 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450117    5972 scope.go:122] "RemoveContainer" containerID="966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a"
Oct 30 18:52:16.450399 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:52:16.450380    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a\": container with ID starting with 966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a not found: ID does not exist" containerID="966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a"
Oct 30 18:52:16.450452 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450410    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a"} err="failed to get container status \"966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a\": rpc error: code = NotFound desc = could not find container \"966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a\": container with ID starting with 966db7af11d93129ecdad214c9d00ffd4bb167e8f5abfc39da3c8fe7952ebc8a not found: ID does not exist"
Oct 30 18:52:16.450452 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450429    5972 scope.go:122] "RemoveContainer" containerID="2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7"
Oct 30 18:52:16.450697 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:52:16.450680    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7\": container with ID starting with 2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7 not found: ID does not exist" containerID="2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7"
Oct 30 18:52:16.450768 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450706    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7"} err="failed to get container status \"2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7\": rpc error: code = NotFound desc = could not find container \"2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7\": container with ID starting with 2138ff665c14372cee272dd6b2e75ee869828c5421d937bf427fa2b32801c1f7 not found: ID does not exist"
Oct 30 18:52:16.450768 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450726    5972 scope.go:122] "RemoveContainer" containerID="d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0"
Oct 30 18:52:16.450966 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:52:16.450950    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0\": container with ID starting with d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0 not found: ID does not exist" containerID="d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0"
Oct 30 18:52:16.451004 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450971    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0"} err="failed to get container status \"d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0\": rpc error: code = NotFound desc = could not find container \"d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0\": container with ID starting with d89df2de0d8b56d55e3ae820c4de10b122460b766164a4a51dd0ff47329b2cb0 not found: ID does not exist"
Oct 30 18:52:16.451004 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.450984    5972 scope.go:122] "RemoveContainer" containerID="fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4"
Oct 30 18:52:16.451197 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:52:16.451178    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4\": container with ID starting with fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4 not found: ID does not exist" containerID="fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4"
Oct 30 18:52:16.451249 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.451205    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4"} err="failed to get container status \"fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4\": rpc error: code = NotFound desc = could not find container \"fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4\": container with ID starting with fab6511471dd3d140c583f44a0768277d1979c1db70461dd5379f3694be490b4 not found: ID does not exist"
Oct 30 18:52:16.627373 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.627303    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"]
Oct 30 18:52:16.675353 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:16.675324    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-predictor-5c97bf4d76-4wb6g"]
Oct 30 18:52:18.343340 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:18.343308    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" path="/var/lib/kubelet/pods/da3e514f-41ba-4792-8064-300bdb6cad13/volumes"
Oct 30 18:52:20.719723 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.719686    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"]
Oct 30 18:52:20.720201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720183    5972 cpu_manager.go:414] "containerMap: removing container" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy"
Oct 30 18:52:20.720201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720200    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy"
Oct 30 18:52:20.720324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720220    5972 cpu_manager.go:414] "containerMap: removing container" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container"
Oct 30 18:52:20.720324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720228    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container"
Oct 30 18:52:20.720324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720245    5972 cpu_manager.go:414] "containerMap: removing container" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent"
Oct 30 18:52:20.720324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720252    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent"
Oct 30 18:52:20.720324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720270    5972 cpu_manager.go:414] "containerMap: removing container" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="storage-initializer"
Oct 30 18:52:20.720324 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720277    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="storage-initializer"
Oct 30 18:52:20.720641 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720361    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kube-rbac-proxy"
Oct 30 18:52:20.720641 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720376    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="kserve-container"
Oct 30 18:52:20.720641 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.720386    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e514f-41ba-4792-8064-300bdb6cad13" containerName="agent"
Oct 30 18:52:20.725407 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.725384    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.740171 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.740148    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.740294 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.740264    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ddt\" (UniqueName: \"kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.740352 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.740293    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.740401 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.740381    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.740461 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.740448    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.775893 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.775859    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 18:52:20.776071 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.776052    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Oct 30 18:52:20.776121 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.776094    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:52:20.776166 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.776120    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 18:52:20.823773 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.823747    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 18:52:20.823773 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.823747    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 18:52:20.823981 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.823837    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Oct 30 18:52:20.824911 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.824891    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"]
Oct 30 18:52:20.841390 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.841365    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ddt\" (UniqueName: \"kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.841506 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.841393    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.841506 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.841421    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.841506 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.841471    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.841684 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.841523    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.841901 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.841877    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.842162 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.842145    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.842240 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.842183    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.844117 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.844100    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:20.901845 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:20.901806    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ddt\" (UniqueName: \"kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt\") pod \"isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:21.044268 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:21.044187    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:21.385878 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:21.385787    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:21.458783 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:21.458755    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"]
Oct 30 18:52:21.461778 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:52:21.461752    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9138d0_0172_4124_96d2_204dac94836c.slice/crio-5e304b07a25356f5b12553f22619d8913ca376e190af5f74a58b5cc022c1591a WatchSource:0}: Error finding container 5e304b07a25356f5b12553f22619d8913ca376e190af5f74a58b5cc022c1591a: Status 404 returned error can't find the container with id 5e304b07a25356f5b12553f22619d8913ca376e190af5f74a58b5cc022c1591a
Oct 30 18:52:22.446602 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:22.446550    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerStarted","Data":"ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90"}
Oct 30 18:52:22.446988 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:22.446657    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerStarted","Data":"5e304b07a25356f5b12553f22619d8913ca376e190af5f74a58b5cc022c1591a"}
Oct 30 18:52:22.869296 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:52:22.869218    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:52:25.460807 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:25.460773    5972 generic.go:356] "Generic (PLEG): container finished" podID="0f9138d0-0172-4124-96d2-204dac94836c" containerID="ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90" exitCode=0
Oct 30 18:52:25.461185 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:25.460833    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerDied","Data":"ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90"}
Oct 30 18:52:26.468036 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.468007    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerStarted","Data":"7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790"}
Oct 30 18:52:26.468426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.468072    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerStarted","Data":"d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816"}
Oct 30 18:52:26.468426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.468125    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerStarted","Data":"7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804"}
Oct 30 18:52:26.468426 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.468402    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:26.468563 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.468547    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:26.469757 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.469733    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:52:26.615396 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:26.615349    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podStartSLOduration=2.777461753 podStartE2EDuration="6.615335402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.837873649s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:26.584430494 +0000 UTC m=+796.376397784" watchObservedRunningTime="2026-10-30 18:52:26.615335402 +0000 UTC m=+796.407302689"
Oct 30 18:52:27.472297 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:27.472263    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:52:27.472803 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:27.472309    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:27.473480 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:27.473457    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:28.477448 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:28.477411    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:52:28.477881 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:28.477859    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:31.386202 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:31.386169    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:33.481325 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:33.481293    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:52:33.481916 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:33.481889    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:52:33.482072 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:33.482046    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:41.385635 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:41.385571    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:43.482649 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:43.482605    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:52:43.483071 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:43.483021    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:51.386452 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:51.386379    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:52:52.869899 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:52:52.869870    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:52:53.482366 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:53.482326    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:52:53.482799 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:52:53.482770    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:01.385958 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:01.385916    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:53:03.482629 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:03.482554    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:53:03.483076 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:03.483055    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:11.386511 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:11.386468    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Oct 30 18:53:13.482430 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:13.482381    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:53:13.482896 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:13.482860    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:21.386348 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:21.386321    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:53:22.870098 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:53:22.870005    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:53:23.481856 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:23.481812    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:53:23.482303 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:23.482278    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:33.481775 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:33.481737    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:53:33.482253 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:33.482134    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:43.482772 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:43.482746    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:53:43.483281 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:43.483035    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:53:44.102098 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.102069    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"]
Oct 30 18:53:44.105804 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.105786    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.156100 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.156079    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-secondary-27eef3-kube-rbac-proxy-sar-config\""
Oct 30 18:53:44.156100 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.156080    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"invalid-s3-sa-27eef3-dockercfg-rcwns\""
Oct 30 18:53:44.156237 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.156116    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"invalid-s3-secret-27eef3\""
Oct 30 18:53:44.156237 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.156127    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-secondary-27eef3-predictor-serving-cert\""
Oct 30 18:53:44.204740 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.204717    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"]
Oct 30 18:53:44.259919 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.259889    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.260033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.259923    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qhb\" (UniqueName: \"kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.260033 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.260004    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.260114 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.260055    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.260114 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.260085    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.360912 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.360851    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.360912 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.360891    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.361056 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.360950    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.361056 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.360981    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qhb\" (UniqueName: \"kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.361056 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.361020    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.361406 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.361385    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.361692 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.361677    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.361746 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.361702    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.363494 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.363479    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.420132 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.420110    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qhb\" (UniqueName: \"kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb\") pod \"isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") " pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.422946 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.422930    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:53:44.836074 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:44.836045    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"]
Oct 30 18:53:44.839347 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:53:44.839323    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafe0faa_daa4_4de1_8f27_c349c825fa91.slice/crio-c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00 WatchSource:0}: Error finding container c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00: Status 404 returned error can't find the container with id c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00
Oct 30 18:53:45.797025 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:45.796994    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" event={"ID":"cafe0faa-daa4-4de1-8f27-c349c825fa91","Type":"ContainerStarted","Data":"73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1"}
Oct 30 18:53:45.797210 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:45.797051    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" event={"ID":"cafe0faa-daa4-4de1-8f27-c349c825fa91","Type":"ContainerStarted","Data":"c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00"}
Oct 30 18:53:51.823798 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:51.823767    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/0.log"
Oct 30 18:53:51.824185 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:51.823803    5972 generic.go:356] "Generic (PLEG): container finished" podID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerID="73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1" exitCode=1
Oct 30 18:53:51.824185 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:51.823882    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" event={"ID":"cafe0faa-daa4-4de1-8f27-c349c825fa91","Type":"ContainerDied","Data":"73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1"}
Oct 30 18:53:52.830027 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:52.829998    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/0.log"
Oct 30 18:53:52.830435 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:52.830123    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" event={"ID":"cafe0faa-daa4-4de1-8f27-c349c825fa91","Type":"ContainerStarted","Data":"bc8d806e392dce2f88e31eff45cd2d9b786c58ae9758d7aeb5fbab1fb0a8bec3"}
Oct 30 18:53:52.869777 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:53:52.869749    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:53:56.217792 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:56.217756    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"]
Oct 30 18:53:56.218303 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:56.218093    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" containerID="cri-o://7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804" gracePeriod=30
Oct 30 18:53:56.218303 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:56.218109    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" containerID="cri-o://7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790" gracePeriod=30
Oct 30 18:53:56.218303 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:56.218131    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" containerID="cri-o://d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816" gracePeriod=30
Oct 30 18:53:56.850258 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:56.850229    5972 generic.go:356] "Generic (PLEG): container finished" podID="0f9138d0-0172-4124-96d2-204dac94836c" containerID="d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816" exitCode=0
Oct 30 18:53:56.850429 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:56.850297    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerDied","Data":"d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816"}
Oct 30 18:53:58.477909 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.477874    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 30 18:53:58.860823 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.860749    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/1.log"
Oct 30 18:53:58.861163 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.861148    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/0.log"
Oct 30 18:53:58.861209 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.861180    5972 generic.go:356] "Generic (PLEG): container finished" podID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerID="bc8d806e392dce2f88e31eff45cd2d9b786c58ae9758d7aeb5fbab1fb0a8bec3" exitCode=1
Oct 30 18:53:58.861288 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.861258    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" event={"ID":"cafe0faa-daa4-4de1-8f27-c349c825fa91","Type":"ContainerDied","Data":"bc8d806e392dce2f88e31eff45cd2d9b786c58ae9758d7aeb5fbab1fb0a8bec3"}
Oct 30 18:53:58.861344 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.861334    5972 scope.go:122] "RemoveContainer" containerID="73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1"
Oct 30 18:53:58.861660 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.861642    5972 scope.go:122] "RemoveContainer" containerID="73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1"
Oct 30 18:53:58.872399 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:53:58.872381    5972 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_e2e-gw1-4a3552942a(cafe0faa-daa4-4de1-8f27-c349c825fa91)\"" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91"
Oct 30 18:53:58.872531 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:53:58.872507    5972 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_e2e-gw1-4a3552942a_cafe0faa-daa4-4de1-8f27-c349c825fa91_0 in pod sandbox c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00 from index: no such id: '73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1'" containerID="73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1"
Oct 30 18:53:58.872632 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:58.872543    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_e2e-gw1-4a3552942a_cafe0faa-daa4-4de1-8f27-c349c825fa91_0 in pod sandbox c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00 from index: no such id: '73f95192683d88bc88f6cd40e3fa7fb3eb803fc3e50da3c0ecac58a67ee411d1'"
Oct 30 18:53:59.867175 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:53:59.867146    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/1.log"
Oct 30 18:54:00.874547 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:00.874513    5972 generic.go:356] "Generic (PLEG): container finished" podID="0f9138d0-0172-4124-96d2-204dac94836c" containerID="7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804" exitCode=0
Oct 30 18:54:00.874963 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:00.874604    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerDied","Data":"7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804"}
Oct 30 18:54:03.477830 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:03.477788    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 30 18:54:03.482146 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:03.482108    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:54:03.482459 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:03.482438    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:06.023779 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.023748    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"]
Oct 30 18:54:06.144299 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.144277    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/1.log"
Oct 30 18:54:06.146689 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.146670    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:54:06.238333 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238298    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb\" (UniqueName: \"kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb\") pod \"cafe0faa-daa4-4de1-8f27-c349c825fa91\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") "
Oct 30 18:54:06.238539 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238382    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location\") pod \"cafe0faa-daa4-4de1-8f27-c349c825fa91\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") "
Oct 30 18:54:06.238539 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238412    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls\") pod \"cafe0faa-daa4-4de1-8f27-c349c825fa91\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") "
Oct 30 18:54:06.238539 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238449    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert\") pod \"cafe0faa-daa4-4de1-8f27-c349c825fa91\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") "
Oct 30 18:54:06.238539 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238499    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config\") pod \"cafe0faa-daa4-4de1-8f27-c349c825fa91\" (UID: \"cafe0faa-daa4-4de1-8f27-c349c825fa91\") "
Oct 30 18:54:06.238764 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238709    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location" pod "cafe0faa-daa4-4de1-8f27-c349c825fa91" (UID: "cafe0faa-daa4-4de1-8f27-c349c825fa91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:06.238939 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238916    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert" pod "cafe0faa-daa4-4de1-8f27-c349c825fa91" (UID: "cafe0faa-daa4-4de1-8f27-c349c825fa91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:06.239018 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.238936    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config" pod "cafe0faa-daa4-4de1-8f27-c349c825fa91" (UID: "cafe0faa-daa4-4de1-8f27-c349c825fa91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:06.240660 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.240635    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls" pod "cafe0faa-daa4-4de1-8f27-c349c825fa91" (UID: "cafe0faa-daa4-4de1-8f27-c349c825fa91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:06.240769 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.240747    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb" pod "cafe0faa-daa4-4de1-8f27-c349c825fa91" (UID: "cafe0faa-daa4-4de1-8f27-c349c825fa91"). InnerVolumeSpecName "kube-api-access-q8qhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:06.303640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.303516    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"]
Oct 30 18:54:06.303884 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.303852    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container" containerID="cri-o://3cc161125c9a4ff61b406f829b661af5f6d3b6736b63dcb8d05627246878722c" gracePeriod=30
Oct 30 18:54:06.304018 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.303879    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kube-rbac-proxy" containerID="cri-o://a3707578c61882cf092512f3fc98a4a8b16b0d5fb880ac47b178a7968b4f5cfa" gracePeriod=30
Oct 30 18:54:06.339330 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.339300    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8qhb\" (UniqueName: \"kubernetes.io/projected/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-api-access-q8qhb\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.339330 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.339335    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe0faa-daa4-4de1-8f27-c349c825fa91-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.339613 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.339346    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe0faa-daa4-4de1-8f27-c349c825fa91-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.339613 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.339355    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.339613 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.339364    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe0faa-daa4-4de1-8f27-c349c825fa91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.381244 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.381206    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Oct 30 18:54:06.902274 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.902243    5972 generic.go:356] "Generic (PLEG): container finished" podID="a17d4300-f366-458c-b8c7-596546159e3a" containerID="a3707578c61882cf092512f3fc98a4a8b16b0d5fb880ac47b178a7968b4f5cfa" exitCode=0
Oct 30 18:54:06.902460 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.902316    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerDied","Data":"a3707578c61882cf092512f3fc98a4a8b16b0d5fb880ac47b178a7968b4f5cfa"}
Oct 30 18:54:06.903669 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.903649    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc_cafe0faa-daa4-4de1-8f27-c349c825fa91/storage-initializer/1.log"
Oct 30 18:54:06.903791 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.903737    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc" event={"ID":"cafe0faa-daa4-4de1-8f27-c349c825fa91","Type":"ContainerDied","Data":"c69bd2fbf5408ad70a4eb73e7a69cdf033eab80a21441d7901dc952c326def00"}
Oct 30 18:54:06.903791 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.903750    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"
Oct 30 18:54:06.903902 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:06.903796    5972 scope.go:122] "RemoveContainer" containerID="bc8d806e392dce2f88e31eff45cd2d9b786c58ae9758d7aeb5fbab1fb0a8bec3"
Oct 30 18:54:07.126440 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:07.126410    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"]
Oct 30 18:54:07.132212 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:07.132191    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-secondary-27eef3-predictor-6cdb8755fc-kxbhc"]
Oct 30 18:54:08.338405 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:08.338373    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" path="/var/lib/kubelet/pods/cafe0faa-daa4-4de1-8f27-c349c825fa91/volumes"
Oct 30 18:54:08.478174 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:08.478142    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 30 18:54:08.478336 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:08.478266    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:54:10.448271 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:10.448244    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:54:10.462196 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:10.462168    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:54:10.923274 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:10.923246    5972 generic.go:356] "Generic (PLEG): container finished" podID="a17d4300-f366-458c-b8c7-596546159e3a" containerID="3cc161125c9a4ff61b406f829b661af5f6d3b6736b63dcb8d05627246878722c" exitCode=0
Oct 30 18:54:10.923405 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:10.923304    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerDied","Data":"3cc161125c9a4ff61b406f829b661af5f6d3b6736b63dcb8d05627246878722c"}
Oct 30 18:54:11.051155 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.051128    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:54:11.184272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184203    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls\") pod \"a17d4300-f366-458c-b8c7-596546159e3a\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") "
Oct 30 18:54:11.184272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184244    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location\") pod \"a17d4300-f366-458c-b8c7-596546159e3a\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") "
Oct 30 18:54:11.184272 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184271    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config\") pod \"a17d4300-f366-458c-b8c7-596546159e3a\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") "
Oct 30 18:54:11.184534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184305    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert\") pod \"a17d4300-f366-458c-b8c7-596546159e3a\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") "
Oct 30 18:54:11.184534 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184348    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg\" (UniqueName: \"kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg\") pod \"a17d4300-f366-458c-b8c7-596546159e3a\" (UID: \"a17d4300-f366-458c-b8c7-596546159e3a\") "
Oct 30 18:54:11.184668 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184535    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location" pod "a17d4300-f366-458c-b8c7-596546159e3a" (UID: "a17d4300-f366-458c-b8c7-596546159e3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:11.184707 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184687    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert" pod "a17d4300-f366-458c-b8c7-596546159e3a" (UID: "a17d4300-f366-458c-b8c7-596546159e3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:11.184707 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.184685    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config" pod "a17d4300-f366-458c-b8c7-596546159e3a" (UID: "a17d4300-f366-458c-b8c7-596546159e3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:11.186457 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.186437    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls" pod "a17d4300-f366-458c-b8c7-596546159e3a" (UID: "a17d4300-f366-458c-b8c7-596546159e3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:11.186594 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.186558    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg" pod "a17d4300-f366-458c-b8c7-596546159e3a" (UID: "a17d4300-f366-458c-b8c7-596546159e3a"). InnerVolumeSpecName "kube-api-access-wtwjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:11.285860 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.285829    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwjg\" (UniqueName: \"kubernetes.io/projected/a17d4300-f366-458c-b8c7-596546159e3a-kube-api-access-wtwjg\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:11.285860 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.285855    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17d4300-f366-458c-b8c7-596546159e3a-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:11.285860 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.285865    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17d4300-f366-458c-b8c7-596546159e3a-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:11.286057 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.285874    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:11.286057 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.285884    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17d4300-f366-458c-b8c7-596546159e3a-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:11.929278 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.929241    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r" event={"ID":"a17d4300-f366-458c-b8c7-596546159e3a","Type":"ContainerDied","Data":"b2a4dbd38a4fa717df1a2e0a47b6fc43b40db691cbaa3c07a60bcd379eab89b2"}
Oct 30 18:54:11.929278 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.929266    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"
Oct 30 18:54:11.929726 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.929327    5972 scope.go:122] "RemoveContainer" containerID="a3707578c61882cf092512f3fc98a4a8b16b0d5fb880ac47b178a7968b4f5cfa"
Oct 30 18:54:11.938476 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.938461    5972 scope.go:122] "RemoveContainer" containerID="3cc161125c9a4ff61b406f829b661af5f6d3b6736b63dcb8d05627246878722c"
Oct 30 18:54:11.945706 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:11.945687    5972 scope.go:122] "RemoveContainer" containerID="05f90cc1dadee10d7b09dee19d48a509706307b6c72ea5eee29524e27bae9f98"
Oct 30 18:54:12.136564 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:12.136532    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"]
Oct 30 18:54:12.184109 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:12.184031    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-primary-27eef3-predictor-5cc55bfff4-9lh5r"]
Oct 30 18:54:12.339548 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:12.339517    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17d4300-f366-458c-b8c7-596546159e3a" path="/var/lib/kubelet/pods/a17d4300-f366-458c-b8c7-596546159e3a/volumes"
Oct 30 18:54:13.477924 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:13.477882    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 30 18:54:13.482146 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:13.482116    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:54:13.482389 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:13.482364    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:15.267067 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267037    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"]
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267366    5972 cpu_manager.go:414] "containerMap: removing container" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerName="storage-initializer"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267376    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerName="storage-initializer"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267402    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267408    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267414    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kube-rbac-proxy"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267420    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kube-rbac-proxy"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267433    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="storage-initializer"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267438    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="storage-initializer"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267491    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerName="storage-initializer"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267500    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kserve-container"
Oct 30 18:54:15.267538 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267505    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17d4300-f366-458c-b8c7-596546159e3a" containerName="kube-rbac-proxy"
Oct 30 18:54:15.267906 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267569    5972 cpu_manager.go:414] "containerMap: removing container" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerName="storage-initializer"
Oct 30 18:54:15.267906 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267574    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerName="storage-initializer"
Oct 30 18:54:15.267906 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.267650    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafe0faa-daa4-4de1-8f27-c349c825fa91" containerName="storage-initializer"
Oct 30 18:54:15.272122 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.272105    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.322853 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.322823    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:54:15.322969 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.322881    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-init-fail-aec7b3-kube-rbac-proxy-sar-config\""
Oct 30 18:54:15.322969 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.322882    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:54:15.370567 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.370543    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"fail-s3-secret-aec7b3\""
Oct 30 18:54:15.370704 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.370604    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:54:15.370704 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.370603    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-init-fail-aec7b3-predictor-serving-cert\""
Oct 30 18:54:15.370704 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.370629    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"fail-s3-sa-aec7b3-dockercfg-rv7xj\""
Oct 30 18:54:15.371685 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.371668    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"]
Oct 30 18:54:15.417768 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.417743    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.417868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.417776    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.417868 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.417801    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.417946 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.417912    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.417946 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.417939    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg84l\" (UniqueName: \"kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519230 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519166    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg84l\" (UniqueName: \"kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519230 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519222    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519379 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519330    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519430 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519374    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519465 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519445    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519691 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519674    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519926 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519910    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.519969 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.519956    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.521965 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.521951    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.578600 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.578555    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg84l\" (UniqueName: \"kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l\") pod \"isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") " pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:15.589439 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:15.589417    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:16.006724 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:16.006694    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"]
Oct 30 18:54:16.009755 ip-10-0-134-133 kubenswrapper[5972]: W1030 18:54:16.009726    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49373ece_8539_4d6a_97d3_ba5432f76d31.slice/crio-e772cb85329aeb351e8b99c8a1c9b84cb43f1bdd4a7b20b086015dd6ff139143 WatchSource:0}: Error finding container e772cb85329aeb351e8b99c8a1c9b84cb43f1bdd4a7b20b086015dd6ff139143: Status 404 returned error can't find the container with id e772cb85329aeb351e8b99c8a1c9b84cb43f1bdd4a7b20b086015dd6ff139143
Oct 30 18:54:16.951640 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:16.951600    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" event={"ID":"49373ece-8539-4d6a-97d3-ba5432f76d31","Type":"ContainerStarted","Data":"e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8"}
Oct 30 18:54:16.952030 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:16.951712    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" event={"ID":"49373ece-8539-4d6a-97d3-ba5432f76d31","Type":"ContainerStarted","Data":"e772cb85329aeb351e8b99c8a1c9b84cb43f1bdd4a7b20b086015dd6ff139143"}
Oct 30 18:54:18.478341 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:18.478252    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 30 18:54:22.869111 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:22.869076    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:54:22.979825 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:22.979800    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/0.log"
Oct 30 18:54:22.979958 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:22.979835    5972 generic.go:356] "Generic (PLEG): container finished" podID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerID="e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8" exitCode=1
Oct 30 18:54:22.979958 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:22.979873    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" event={"ID":"49373ece-8539-4d6a-97d3-ba5432f76d31","Type":"ContainerDied","Data":"e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8"}
Oct 30 18:54:23.477803 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.477759    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Oct 30 18:54:23.482047 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.482021    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Oct 30 18:54:23.482174 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.482146    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:54:23.482445 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.482407    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:23.482546 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.482531    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:54:23.986278 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.986252    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/0.log"
Oct 30 18:54:23.986688 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:23.986325    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" event={"ID":"49373ece-8539-4d6a-97d3-ba5432f76d31","Type":"ContainerStarted","Data":"b1b3ae1eb74c486a65e961c1642858536042e6fd2266b9e7fbdf1d1a90720ca8"}
Oct 30 18:54:26.376765 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.376742    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:54:26.522733 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.522647    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert\") pod \"0f9138d0-0172-4124-96d2-204dac94836c\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") "
Oct 30 18:54:26.522733 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.522708    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls\") pod \"0f9138d0-0172-4124-96d2-204dac94836c\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") "
Oct 30 18:54:26.522995 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.522741    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location\") pod \"0f9138d0-0172-4124-96d2-204dac94836c\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") "
Oct 30 18:54:26.522995 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.522778    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt\" (UniqueName: \"kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt\") pod \"0f9138d0-0172-4124-96d2-204dac94836c\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") "
Oct 30 18:54:26.522995 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.522802    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config\") pod \"0f9138d0-0172-4124-96d2-204dac94836c\" (UID: \"0f9138d0-0172-4124-96d2-204dac94836c\") "
Oct 30 18:54:26.523158 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.523111    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location" pod "0f9138d0-0172-4124-96d2-204dac94836c" (UID: "0f9138d0-0172-4124-96d2-204dac94836c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:26.523200 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.523061    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert" pod "0f9138d0-0172-4124-96d2-204dac94836c" (UID: "0f9138d0-0172-4124-96d2-204dac94836c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:26.523236 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.523198    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config" pod "0f9138d0-0172-4124-96d2-204dac94836c" (UID: "0f9138d0-0172-4124-96d2-204dac94836c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:26.525201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.525174    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls" pod "0f9138d0-0172-4124-96d2-204dac94836c" (UID: "0f9138d0-0172-4124-96d2-204dac94836c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:26.525201 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.525173    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt" pod "0f9138d0-0172-4124-96d2-204dac94836c" (UID: "0f9138d0-0172-4124-96d2-204dac94836c"). InnerVolumeSpecName "kube-api-access-l8ddt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:26.623863 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.623826    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ddt\" (UniqueName: \"kubernetes.io/projected/0f9138d0-0172-4124-96d2-204dac94836c-kube-api-access-l8ddt\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:26.623863 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.623856    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:26.623863 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.623868    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f9138d0-0172-4124-96d2-204dac94836c-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:26.624105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.623878    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f9138d0-0172-4124-96d2-204dac94836c-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:26.624105 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:26.623886    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f9138d0-0172-4124-96d2-204dac94836c-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:27.001984 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.001953    5972 generic.go:356] "Generic (PLEG): container finished" podID="0f9138d0-0172-4124-96d2-204dac94836c" containerID="7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790" exitCode=0
Oct 30 18:54:27.002178 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.002028    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"
Oct 30 18:54:27.002178 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.002026    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerDied","Data":"7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790"}
Oct 30 18:54:27.002178 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.002165    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8" event={"ID":"0f9138d0-0172-4124-96d2-204dac94836c","Type":"ContainerDied","Data":"5e304b07a25356f5b12553f22619d8913ca376e190af5f74a58b5cc022c1591a"}
Oct 30 18:54:27.002338 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.002200    5972 scope.go:122] "RemoveContainer" containerID="7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790"
Oct 30 18:54:27.003838 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.003819    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/1.log"
Oct 30 18:54:27.004217 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.004202    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/0.log"
Oct 30 18:54:27.004290 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.004232    5972 generic.go:356] "Generic (PLEG): container finished" podID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerID="b1b3ae1eb74c486a65e961c1642858536042e6fd2266b9e7fbdf1d1a90720ca8" exitCode=1
Oct 30 18:54:27.004347 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.004300    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" event={"ID":"49373ece-8539-4d6a-97d3-ba5432f76d31","Type":"ContainerDied","Data":"b1b3ae1eb74c486a65e961c1642858536042e6fd2266b9e7fbdf1d1a90720ca8"}
Oct 30 18:54:27.004678 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.004663    5972 scope.go:122] "RemoveContainer" containerID="e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8"
Oct 30 18:54:27.011412 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.011381    5972 scope.go:122] "RemoveContainer" containerID="d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816"
Oct 30 18:54:27.015689 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:27.015669    5972 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_e2e-gw1-4a3552942a(49373ece-8539-4d6a-97d3-ba5432f76d31)\"" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31"
Oct 30 18:54:27.019029 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.019012    5972 scope.go:122] "RemoveContainer" containerID="7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804"
Oct 30 18:54:27.025940 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.025926    5972 scope.go:122] "RemoveContainer" containerID="ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90"
Oct 30 18:54:27.033187 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.033168    5972 scope.go:122] "RemoveContainer" containerID="7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790"
Oct 30 18:54:27.033455 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:27.033435    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790\": container with ID starting with 7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790 not found: ID does not exist" containerID="7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790"
Oct 30 18:54:27.033510 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.033461    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790"} err="failed to get container status \"7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790\": rpc error: code = NotFound desc = could not find container \"7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790\": container with ID starting with 7652b3f1693e192d4379ae5b367212bb44706743685aade57b8f482c3fdf2790 not found: ID does not exist"
Oct 30 18:54:27.033510 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.033477    5972 scope.go:122] "RemoveContainer" containerID="d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816"
Oct 30 18:54:27.033753 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:27.033736    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816\": container with ID starting with d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816 not found: ID does not exist" containerID="d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816"
Oct 30 18:54:27.033805 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.033757    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816"} err="failed to get container status \"d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816\": rpc error: code = NotFound desc = could not find container \"d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816\": container with ID starting with d6d8766dcaa16c5cb7ea26b5d465d9e01ef226746b8bc986fef3a4f5dc438816 not found: ID does not exist"
Oct 30 18:54:27.033805 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.033779    5972 scope.go:122] "RemoveContainer" containerID="7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804"
Oct 30 18:54:27.034012 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:27.033999    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804\": container with ID starting with 7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804 not found: ID does not exist" containerID="7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804"
Oct 30 18:54:27.034058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.034015    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804"} err="failed to get container status \"7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804\": rpc error: code = NotFound desc = could not find container \"7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804\": container with ID starting with 7dc1a874914a76c5ad4a4b3ed5184b9181cf77177f54b0d486f172e9871ea804 not found: ID does not exist"
Oct 30 18:54:27.034058 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.034037    5972 scope.go:122] "RemoveContainer" containerID="ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90"
Oct 30 18:54:27.034301 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:27.034283    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90\": container with ID starting with ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90 not found: ID does not exist" containerID="ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90"
Oct 30 18:54:27.034353 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.034308    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90"} err="failed to get container status \"ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90\": rpc error: code = NotFound desc = could not find container \"ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90\": container with ID starting with ca826e32d0d5f11df32a0421053165efa55a1ad8195901bcfb50a122a1e9fe90 not found: ID does not exist"
Oct 30 18:54:27.034353 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.034326    5972 scope.go:122] "RemoveContainer" containerID="e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8"
Oct 30 18:54:27.034554 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:27.034539    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8\": container with ID starting with e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8 not found: ID does not exist" containerID="e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8"
Oct 30 18:54:27.034878 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.034560    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8"} err="failed to get container status \"e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8\": rpc error: code = NotFound desc = could not find container \"e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8\": container with ID starting with e50be2d8e22fce438c85115169d4ea97f959ec236349e3a0589f34083feef1f8 not found: ID does not exist"
Oct 30 18:54:27.277400 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.277320    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"]
Oct 30 18:54:27.283633 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:27.283610    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-batcher-custom-predictor-546559c898-qgrr8"]
Oct 30 18:54:28.011965 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:28.011939    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/1.log"
Oct 30 18:54:28.338839 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:28.338772    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9138d0-0172-4124-96d2-204dac94836c" path="/var/lib/kubelet/pods/0f9138d0-0172-4124-96d2-204dac94836c/volumes"
Oct 30 18:54:29.951456 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:29.951422    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"]
Oct 30 18:54:30.073526 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.073323    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/1.log"
Oct 30 18:54:30.076833 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.076811    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:30.252266 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252195    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l\" (UniqueName: \"kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l\") pod \"49373ece-8539-4d6a-97d3-ba5432f76d31\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") "
Oct 30 18:54:30.252266 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252228    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config\") pod \"49373ece-8539-4d6a-97d3-ba5432f76d31\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") "
Oct 30 18:54:30.252456 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252351    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls\" (UniqueName: \"kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls\") pod \"49373ece-8539-4d6a-97d3-ba5432f76d31\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") "
Oct 30 18:54:30.252520 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252453    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location\") pod \"49373ece-8539-4d6a-97d3-ba5432f76d31\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") "
Oct 30 18:54:30.252520 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252484    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert\") pod \"49373ece-8539-4d6a-97d3-ba5432f76d31\" (UID: \"49373ece-8539-4d6a-97d3-ba5432f76d31\") "
Oct 30 18:54:30.252653 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252610    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config" pod "49373ece-8539-4d6a-97d3-ba5432f76d31" (UID: "49373ece-8539-4d6a-97d3-ba5432f76d31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:30.252775 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252753    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:30.252775 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252749    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location" pod "49373ece-8539-4d6a-97d3-ba5432f76d31" (UID: "49373ece-8539-4d6a-97d3-ba5432f76d31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:30.252944 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.252843    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert" pod "49373ece-8539-4d6a-97d3-ba5432f76d31" (UID: "49373ece-8539-4d6a-97d3-ba5432f76d31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:30.254413 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.254389    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l" pod "49373ece-8539-4d6a-97d3-ba5432f76d31" (UID: "49373ece-8539-4d6a-97d3-ba5432f76d31"). InnerVolumeSpecName "kube-api-access-dg84l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:30.254493 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.254465    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls" pod "49373ece-8539-4d6a-97d3-ba5432f76d31" (UID: "49373ece-8539-4d6a-97d3-ba5432f76d31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:30.353732 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.353708    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49373ece-8539-4d6a-97d3-ba5432f76d31-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:30.353732 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.353730    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49373ece-8539-4d6a-97d3-ba5432f76d31-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:30.353858 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.353740    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg84l\" (UniqueName: \"kubernetes.io/projected/49373ece-8539-4d6a-97d3-ba5432f76d31-kube-api-access-dg84l\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:30.353858 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:30.353749    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49373ece-8539-4d6a-97d3-ba5432f76d31-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:31.027494 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:31.027464    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k_49373ece-8539-4d6a-97d3-ba5432f76d31/storage-initializer/1.log"
Oct 30 18:54:31.027976 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:31.027557    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"
Oct 30 18:54:31.027976 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:31.027571    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k" event={"ID":"49373ece-8539-4d6a-97d3-ba5432f76d31","Type":"ContainerDied","Data":"e772cb85329aeb351e8b99c8a1c9b84cb43f1bdd4a7b20b086015dd6ff139143"}
Oct 30 18:54:31.027976 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:31.027666    5972 scope.go:122] "RemoveContainer" containerID="b1b3ae1eb74c486a65e961c1642858536042e6fd2266b9e7fbdf1d1a90720ca8"
Oct 30 18:54:31.325693 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:31.325615    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"]
Oct 30 18:54:31.329281 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:31.329258    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-init-fail-aec7b3-predictor-6f589c6b9c-j8v5k"]
Oct 30 18:54:32.338956 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:54:32.338923    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" path="/var/lib/kubelet/pods/49373ece-8539-4d6a-97d3-ba5432f76d31/volumes"
Oct 30 18:54:52.870219 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:54:52.870171    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:55:22.869740 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:55:22.869700    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:55:52.882932 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:55:52.882856    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:56:22.869900 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:56:22.869860    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:56:52.869504 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:56:52.869462    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:57:22.869044 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:57:22.868971    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:57:52.869926 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:57:52.869891    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:58:22.869991 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:58:22.869956    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:58:52.869760 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:58:52.869670    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:59:10.480408 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:59:10.480379    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:59:10.490296 ip-10-0-134-133 kubenswrapper[5972]: I1030 18:59:10.490275    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 18:59:22.869718 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:59:22.869685    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 18:59:52.869632 ip-10-0-134-133 kubenswrapper[5972]: E1030 18:59:52.869598    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:00:22.869883 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:00:22.869800    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:00:52.869296 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:00:52.869259    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:01:22.869861 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:01:22.869823    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:01:52.869780 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:01:52.869704    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:02:10.081514 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.081478    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-2eaaca7c"]
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.081966    5972 cpu_manager.go:414] "containerMap: removing container" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="storage-initializer"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.081986    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="storage-initializer"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082011    5972 cpu_manager.go:414] "containerMap: removing container" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082019    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082033    5972 cpu_manager.go:414] "containerMap: removing container" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082041    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082052    5972 cpu_manager.go:414] "containerMap: removing container" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerName="storage-initializer"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082062    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerName="storage-initializer"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082077    5972 cpu_manager.go:414] "containerMap: removing container" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container"
Oct 30 19:02:10.082149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082084    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container"
Oct 30 19:02:10.082650 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082163    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="agent"
Oct 30 19:02:10.082650 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082174    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kube-rbac-proxy"
Oct 30 19:02:10.082650 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082183    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerName="storage-initializer"
Oct 30 19:02:10.082650 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082193    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9138d0-0172-4124-96d2-204dac94836c" containerName="kserve-container"
Oct 30 19:02:10.082650 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.082203    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerName="storage-initializer"
Oct 30 19:02:10.085419 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.085400    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:10.136886 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.136860    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-2eaaca7c"]
Oct 30 19:02:10.183836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.183810    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 19:02:10.183935 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.183839    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 19:02:10.183935 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.183855    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 19:02:10.234310 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.234282    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfnx\" (UniqueName: \"kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx\") pod \"curl-metrics-2eaaca7c\" (UID: \"4eb0fba5-4eae-4acd-8844-9a78b201255f\") " pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:10.335283 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.335216    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfnx\" (UniqueName: \"kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx\") pod \"curl-metrics-2eaaca7c\" (UID: \"4eb0fba5-4eae-4acd-8844-9a78b201255f\") " pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:10.441370 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.441351    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 19:02:10.491626 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.491605    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 19:02:10.494868 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.494852    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfnx\" (UniqueName: \"kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx\") pod \"curl-metrics-2eaaca7c\" (UID: \"4eb0fba5-4eae-4acd-8844-9a78b201255f\") " pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:10.759811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.759784    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 19:02:10.760820 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:10.760805    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:11.125874 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:11.125795    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-2eaaca7c"]
Oct 30 19:02:11.129341 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:02:11.129315    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eb0fba5_4eae_4acd_8844_9a78b201255f.slice/crio-e959aff1a582e21c110ba567e044e75c1f50c79861f4bb95a0b1526d60696591 WatchSource:0}: Error finding container e959aff1a582e21c110ba567e044e75c1f50c79861f4bb95a0b1526d60696591: Status 404 returned error can't find the container with id e959aff1a582e21c110ba567e044e75c1f50c79861f4bb95a0b1526d60696591
Oct 30 19:02:11.131560 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:11.131541    5972 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:02:11.792485 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:11.792411    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-2eaaca7c" event={"ID":"4eb0fba5-4eae-4acd-8844-9a78b201255f","Type":"ContainerStarted","Data":"e959aff1a582e21c110ba567e044e75c1f50c79861f4bb95a0b1526d60696591"}
Oct 30 19:02:13.801745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:13.801706    5972 generic.go:356] "Generic (PLEG): container finished" podID="4eb0fba5-4eae-4acd-8844-9a78b201255f" containerID="7d3a84d3975f9291c88f5da0ec89275b46590f220f6c111c331948896d51849c" exitCode=0
Oct 30 19:02:13.802124 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:13.801776    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-2eaaca7c" event={"ID":"4eb0fba5-4eae-4acd-8844-9a78b201255f","Type":"ContainerDied","Data":"7d3a84d3975f9291c88f5da0ec89275b46590f220f6c111c331948896d51849c"}
Oct 30 19:02:14.921576 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:14.921549    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:15.074083 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.074005    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx\" (UniqueName: \"kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx\") pod \"4eb0fba5-4eae-4acd-8844-9a78b201255f\" (UID: \"4eb0fba5-4eae-4acd-8844-9a78b201255f\") "
Oct 30 19:02:15.076116 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.076085    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx" pod "4eb0fba5-4eae-4acd-8844-9a78b201255f" (UID: "4eb0fba5-4eae-4acd-8844-9a78b201255f"). InnerVolumeSpecName "kube-api-access-5mfnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:15.175110 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.175088    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfnx\" (UniqueName: \"kubernetes.io/projected/4eb0fba5-4eae-4acd-8844-9a78b201255f-kube-api-access-5mfnx\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:15.257016 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.256992    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-2eaaca7c_4eb0fba5-4eae-4acd-8844-9a78b201255f/curl-metrics-2eaaca7c/0.log"
Oct 30 19:02:15.376682 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.376616    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-2eaaca7c"]
Oct 30 19:02:15.380859 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.380837    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-2eaaca7c"]
Oct 30 19:02:15.810774 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.810747    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:15.810774 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.810770    5972 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e959aff1a582e21c110ba567e044e75c1f50c79861f4bb95a0b1526d60696591"
Oct 30 19:02:15.867272 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:02:15.867246    5972 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-2eaaca7c\" is forbidden: User \"system:node:ip-10-0-134-133.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-134-133.us-west-2.compute.internal' and this object" podUID="4eb0fba5-4eae-4acd-8844-9a78b201255f" pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:15.982468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.982441    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-8287feea"]
Oct 30 19:02:15.982817 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.982794    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4eb0fba5-4eae-4acd-8844-9a78b201255f" containerName="curl-metrics-2eaaca7c"
Oct 30 19:02:15.982817 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.982805    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb0fba5-4eae-4acd-8844-9a78b201255f" containerName="curl-metrics-2eaaca7c"
Oct 30 19:02:15.982934 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.982827    5972 cpu_manager.go:414] "containerMap: removing container" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerName="storage-initializer"
Oct 30 19:02:15.982934 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.982833    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49373ece-8539-4d6a-97d3-ba5432f76d31" containerName="storage-initializer"
Oct 30 19:02:15.982934 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.982893    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb0fba5-4eae-4acd-8844-9a78b201255f" containerName="curl-metrics-2eaaca7c"
Oct 30 19:02:15.985653 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:15.985636    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:16.034990 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:02:16.034963    5972 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-2eaaca7c\" is forbidden: User \"system:node:ip-10-0-134-133.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-134-133.us-west-2.compute.internal' and this object" podUID="4eb0fba5-4eae-4acd-8844-9a78b201255f" pod="kserve/curl-metrics-2eaaca7c"
Oct 30 19:02:16.037795 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.037757    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 19:02:16.037795 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.037777    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 19:02:16.037968 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.037779    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 19:02:16.038434 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.038414    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-8287feea"]
Oct 30 19:02:16.082271 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.082214    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbtts\" (UniqueName: \"kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts\") pod \"curl-metrics-8287feea\" (UID: \"78b2bb41-7678-4f33-b629-55ab966689bf\") " pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:16.182670 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.182642    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtts\" (UniqueName: \"kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts\") pod \"curl-metrics-8287feea\" (UID: \"78b2bb41-7678-4f33-b629-55ab966689bf\") " pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:16.242276 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.242248    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtts\" (UniqueName: \"kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts\") pod \"curl-metrics-8287feea\" (UID: \"78b2bb41-7678-4f33-b629-55ab966689bf\") " pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:16.308395 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.308346    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:16.339282 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.339213    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb0fba5-4eae-4acd-8844-9a78b201255f" path="/var/lib/kubelet/pods/4eb0fba5-4eae-4acd-8844-9a78b201255f/volumes"
Oct 30 19:02:16.676669 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.676538    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-8287feea"]
Oct 30 19:02:16.679377 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:02:16.679350    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78b2bb41_7678_4f33_b629_55ab966689bf.slice/crio-205a122e7e534959c20ebdb8891959966393ae7479a3fe4c1d34832347ec4fca WatchSource:0}: Error finding container 205a122e7e534959c20ebdb8891959966393ae7479a3fe4c1d34832347ec4fca: Status 404 returned error can't find the container with id 205a122e7e534959c20ebdb8891959966393ae7479a3fe4c1d34832347ec4fca
Oct 30 19:02:16.816909 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.816880    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-8287feea" event={"ID":"78b2bb41-7678-4f33-b629-55ab966689bf","Type":"ContainerStarted","Data":"fa5ea902aec4bb1b5a60336dfae3f7154cd87d26a97c920787ea7ecc71ae898d"}
Oct 30 19:02:16.817059 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.816918    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-8287feea" event={"ID":"78b2bb41-7678-4f33-b629-55ab966689bf","Type":"ContainerStarted","Data":"205a122e7e534959c20ebdb8891959966393ae7479a3fe4c1d34832347ec4fca"}
Oct 30 19:02:16.927221 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:16.927138    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/curl-metrics-8287feea" podStartSLOduration=1.927124598 podStartE2EDuration="1.927124598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:16.92701698 +0000 UTC m=+1386.718984268" watchObservedRunningTime="2026-10-30 19:02:16.927124598 +0000 UTC m=+1386.719091863"
Oct 30 19:02:17.240875 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:17.240853    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-8287feea_78b2bb41-7678-4f33-b629-55ab966689bf/curl-metrics-8287feea/0.log"
Oct 30 19:02:17.822254 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:17.822224    5972 generic.go:356] "Generic (PLEG): container finished" podID="78b2bb41-7678-4f33-b629-55ab966689bf" containerID="fa5ea902aec4bb1b5a60336dfae3f7154cd87d26a97c920787ea7ecc71ae898d" exitCode=0
Oct 30 19:02:17.822457 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:17.822289    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-8287feea" event={"ID":"78b2bb41-7678-4f33-b629-55ab966689bf","Type":"ContainerDied","Data":"fa5ea902aec4bb1b5a60336dfae3f7154cd87d26a97c920787ea7ecc71ae898d"}
Oct 30 19:02:18.945769 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:18.945747    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:19.004373 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.004350    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts\" (UniqueName: \"kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts\") pod \"78b2bb41-7678-4f33-b629-55ab966689bf\" (UID: \"78b2bb41-7678-4f33-b629-55ab966689bf\") "
Oct 30 19:02:19.010471 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.007206    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts" pod "78b2bb41-7678-4f33-b629-55ab966689bf" (UID: "78b2bb41-7678-4f33-b629-55ab966689bf"). InnerVolumeSpecName "kube-api-access-tbtts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:19.092207 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.092144    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-8287feea"]
Oct 30 19:02:19.103707 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.103685    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-8287feea"]
Oct 30 19:02:19.105389 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.105369    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbtts\" (UniqueName: \"kubernetes.io/projected/78b2bb41-7678-4f33-b629-55ab966689bf-kube-api-access-tbtts\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:19.831639 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.831614    5972 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="205a122e7e534959c20ebdb8891959966393ae7479a3fe4c1d34832347ec4fca"
Oct 30 19:02:19.831812 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:19.831644    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:19.888071 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:02:19.888034    5972 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-8287feea\" is forbidden: User \"system:node:ip-10-0-134-133.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-134-133.us-west-2.compute.internal' and this object" podUID="78b2bb41-7678-4f33-b629-55ab966689bf" pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:20.339009 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:20.338978    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b2bb41-7678-4f33-b629-55ab966689bf" path="/var/lib/kubelet/pods/78b2bb41-7678-4f33-b629-55ab966689bf/volumes"
Oct 30 19:02:20.384957 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:02:20.384925    5972 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-8287feea\" is forbidden: User \"system:node:ip-10-0-134-133.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-134-133.us-west-2.compute.internal' and this object" podUID="78b2bb41-7678-4f33-b629-55ab966689bf" pod="kserve/curl-metrics-8287feea"
Oct 30 19:02:21.660793 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.660763    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"]
Oct 30 19:02:21.661170 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.661065    5972 cpu_manager.go:414] "containerMap: removing container" podUID="78b2bb41-7678-4f33-b629-55ab966689bf" containerName="curl-metrics-8287feea"
Oct 30 19:02:21.661170 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.661074    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b2bb41-7678-4f33-b629-55ab966689bf" containerName="curl-metrics-8287feea"
Oct 30 19:02:21.661170 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.661155    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b2bb41-7678-4f33-b629-55ab966689bf" containerName="curl-metrics-8287feea"
Oct 30 19:02:21.664426 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.664407    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.715283 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.715259    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:02:21.721624 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.721600    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcp6l\" (UniqueName: \"kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.721733 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.721638    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.721733 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.721719    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.721826 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.721754    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.721826 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.721818    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.762927 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.762903    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:02:21.763145 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.762933    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 19:02:21.763246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.762955    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:02:21.763246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.763219    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:02:21.763355 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.763301    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:02:21.763430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.763415    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-predictor-serving-cert\""
Oct 30 19:02:21.764138 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.764119    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"]
Oct 30 19:02:21.822573 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.822549    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.822696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.822640    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcp6l\" (UniqueName: \"kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.822696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.822667    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.822696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.822691    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.822803 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.822773    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.822896 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.822877    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.823449 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.823379    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.823449 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.823379    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.825569 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.825537    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.882057 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.882038    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcp6l\" (UniqueName: \"kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l\") pod \"isvc-sklearn-predictor-56fbdf5dc7-drkbh\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:21.981807 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:21.981779    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:22.396246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:22.396184    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"]
Oct 30 19:02:22.399100 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:02:22.399045    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9578b5a7_d869_4370_8162_a6483fec8f2a.slice/crio-3525bf9829400a2246d14c30255ef6dd5bc07bb80f8f1929ecf78a5b235a0b8b WatchSource:0}: Error finding container 3525bf9829400a2246d14c30255ef6dd5bc07bb80f8f1929ecf78a5b235a0b8b: Status 404 returned error can't find the container with id 3525bf9829400a2246d14c30255ef6dd5bc07bb80f8f1929ecf78a5b235a0b8b
Oct 30 19:02:22.846049 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:22.846014    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerStarted","Data":"4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9"}
Oct 30 19:02:22.846479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:22.846077    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerStarted","Data":"3525bf9829400a2246d14c30255ef6dd5bc07bb80f8f1929ecf78a5b235a0b8b"}
Oct 30 19:02:22.870354 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:02:22.870321    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:02:26.861656 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:26.861619    5972 generic.go:356] "Generic (PLEG): container finished" podID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerID="4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9" exitCode=0
Oct 30 19:02:26.862206 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:26.861682    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerDied","Data":"4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9"}
Oct 30 19:02:27.867540 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:27.867508    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerStarted","Data":"3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69"}
Oct 30 19:02:27.867941 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:27.867569    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerStarted","Data":"badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17"}
Oct 30 19:02:27.867941 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:27.867760    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:27.983819 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:27.983774    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podStartSLOduration=3.22711643 podStartE2EDuration="6.983761158s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756644728s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:27.983406867 +0000 UTC m=+1397.775374137" watchObservedRunningTime="2026-10-30 19:02:27.983761158 +0000 UTC m=+1397.775728424"
Oct 30 19:02:28.871849 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:28.871820    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:28.873083 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:28.873059    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:02:29.875557 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:29.875514    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:02:34.880012 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:34.879979    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:02:34.880497 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:34.880471    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:02:44.880634 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:44.880576    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:02:52.871475 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:02:52.871437    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:02:54.881380 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:02:54.881342    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:03:04.880622 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:04.880565    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:03:14.880438 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:14.880392    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:03:22.869311 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:03:22.869227    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:03:24.881126 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:24.881086    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:03:34.880745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:34.880710    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:03:44.881741 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:44.881712    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:03:50.873859 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:50.873822    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:03:52.869683 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:03:52.869638    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:03:52.994477 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:52.994441    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:03:55.066710 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:55.066685    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:03:57.137835 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:57.137793    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:03:59.210232 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:03:59.210202    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:01.282577 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:01.282553    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:03.354828 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:03.354787    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:05.475217 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:05.475186    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:07.546289 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:07.546256    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:09.617987 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:09.617953    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:10.508206 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:10.508176    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:04:10.519001 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:10.518981    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:04:11.690231 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:11.690201    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:13.762766 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:13.762741    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:15.834312 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:15.834283    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:17.906924 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:17.906880    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:19.978507 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:19.978483    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:22.050215 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:22.050189    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-predictor-56fbdf5dc7-drkbh_9578b5a7-d869-4370-8162-a6483fec8f2a/kserve-container/0.log"
Oct 30 19:04:22.869468 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:04:22.869429    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:04:23.427033 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:23.426994    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"]
Oct 30 19:04:23.427636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:23.427392    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" containerID="cri-o://badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17" gracePeriod=30
Oct 30 19:04:23.427753 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:23.427726    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kube-rbac-proxy" containerID="cri-o://3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69" gracePeriod=30
Oct 30 19:04:24.325727 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:24.325696    5972 generic.go:356] "Generic (PLEG): container finished" podID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerID="3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69" exitCode=0
Oct 30 19:04:24.325884 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:24.325769    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerDied","Data":"3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69"}
Oct 30 19:04:24.876117 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:24.876074    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Oct 30 19:04:24.880432 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:24.880409    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Oct 30 19:04:28.176406 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.176385    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:04:28.259850 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.259798    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config\") pod \"9578b5a7-d869-4370-8162-a6483fec8f2a\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") "
Oct 30 19:04:28.260066 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.259873    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l\" (UniqueName: \"kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l\") pod \"9578b5a7-d869-4370-8162-a6483fec8f2a\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") "
Oct 30 19:04:28.260066 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.259904    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls\") pod \"9578b5a7-d869-4370-8162-a6483fec8f2a\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") "
Oct 30 19:04:28.260066 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.259934    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert\") pod \"9578b5a7-d869-4370-8162-a6483fec8f2a\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") "
Oct 30 19:04:28.260066 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.260015    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location\") pod \"9578b5a7-d869-4370-8162-a6483fec8f2a\" (UID: \"9578b5a7-d869-4370-8162-a6483fec8f2a\") "
Oct 30 19:04:28.260287 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.260232    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config" pod "9578b5a7-d869-4370-8162-a6483fec8f2a" (UID: "9578b5a7-d869-4370-8162-a6483fec8f2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:28.260332 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.260312    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert" pod "9578b5a7-d869-4370-8162-a6483fec8f2a" (UID: "9578b5a7-d869-4370-8162-a6483fec8f2a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:28.260332 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.260311    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location" pod "9578b5a7-d869-4370-8162-a6483fec8f2a" (UID: "9578b5a7-d869-4370-8162-a6483fec8f2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:28.262173 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.262147    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l" pod "9578b5a7-d869-4370-8162-a6483fec8f2a" (UID: "9578b5a7-d869-4370-8162-a6483fec8f2a"). InnerVolumeSpecName "kube-api-access-dcp6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:28.262267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.262221    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls" pod "9578b5a7-d869-4370-8162-a6483fec8f2a" (UID: "9578b5a7-d869-4370-8162-a6483fec8f2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:28.353344 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.353310    5972 generic.go:356] "Generic (PLEG): container finished" podID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerID="badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17" exitCode=0
Oct 30 19:04:28.353475 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.353438    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"
Oct 30 19:04:28.353475 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.353445    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerDied","Data":"badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17"}
Oct 30 19:04:28.353623 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.353545    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh" event={"ID":"9578b5a7-d869-4370-8162-a6483fec8f2a","Type":"ContainerDied","Data":"3525bf9829400a2246d14c30255ef6dd5bc07bb80f8f1929ecf78a5b235a0b8b"}
Oct 30 19:04:28.353623 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.353562    5972 scope.go:122] "RemoveContainer" containerID="3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69"
Oct 30 19:04:28.360778 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.360754    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9578b5a7-d869-4370-8162-a6483fec8f2a-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:28.360870 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.360784    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:28.360870 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.360802    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcp6l\" (UniqueName: \"kubernetes.io/projected/9578b5a7-d869-4370-8162-a6483fec8f2a-kube-api-access-dcp6l\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:28.360870 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.360818    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578b5a7-d869-4370-8162-a6483fec8f2a-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:28.360870 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.360832    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9578b5a7-d869-4370-8162-a6483fec8f2a-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:28.365559 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.365474    5972 scope.go:122] "RemoveContainer" containerID="badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17"
Oct 30 19:04:28.373405 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.373368    5972 scope.go:122] "RemoveContainer" containerID="4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9"
Oct 30 19:04:28.381215 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.381196    5972 scope.go:122] "RemoveContainer" containerID="3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69"
Oct 30 19:04:28.381477 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:04:28.381460    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69\": container with ID starting with 3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69 not found: ID does not exist" containerID="3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69"
Oct 30 19:04:28.381543 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.381485    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69"} err="failed to get container status \"3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69\": rpc error: code = NotFound desc = could not find container \"3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69\": container with ID starting with 3ade26e4964b704002887b82a5f07c23f4ca827aa393c054bb869d4eb08cfe69 not found: ID does not exist"
Oct 30 19:04:28.381543 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.381501    5972 scope.go:122] "RemoveContainer" containerID="badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17"
Oct 30 19:04:28.381739 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:04:28.381723    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17\": container with ID starting with badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17 not found: ID does not exist" containerID="badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17"
Oct 30 19:04:28.381779 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.381754    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17"} err="failed to get container status \"badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17\": rpc error: code = NotFound desc = could not find container \"badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17\": container with ID starting with badf8015e48911b6af5f62463b8d50791c0a6444b255b3a5f07c6e6c38312c17 not found: ID does not exist"
Oct 30 19:04:28.381779 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.381769    5972 scope.go:122] "RemoveContainer" containerID="4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9"
Oct 30 19:04:28.381994 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:04:28.381980    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9\": container with ID starting with 4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9 not found: ID does not exist" containerID="4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9"
Oct 30 19:04:28.382034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.381999    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9"} err="failed to get container status \"4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9\": rpc error: code = NotFound desc = could not find container \"4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9\": container with ID starting with 4f13ee8d9b8ea2ca8c4c3b5382059988a8b1a16705008ac9e1385d1a769108c9 not found: ID does not exist"
Oct 30 19:04:28.512165 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.512092    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"]
Oct 30 19:04:28.516045 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:28.516020    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-predictor-56fbdf5dc7-drkbh"]
Oct 30 19:04:30.338409 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:04:30.338379    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" path="/var/lib/kubelet/pods/9578b5a7-d869-4370-8162-a6483fec8f2a/volumes"
Oct 30 19:04:52.870198 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:04:52.870115    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:05:22.869867 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:05:22.869830    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:05:52.870172 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:05:52.870134    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:05:59.186737 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.186698    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"]
Oct 30 19:05:59.187229 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187161    5972 cpu_manager.go:414] "containerMap: removing container" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="storage-initializer"
Oct 30 19:05:59.187229 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187178    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="storage-initializer"
Oct 30 19:05:59.187229 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187203    5972 cpu_manager.go:414] "containerMap: removing container" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kube-rbac-proxy"
Oct 30 19:05:59.187229 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187211    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kube-rbac-proxy"
Oct 30 19:05:59.187448 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187242    5972 cpu_manager.go:414] "containerMap: removing container" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container"
Oct 30 19:05:59.187448 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187250    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container"
Oct 30 19:05:59.187448 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187352    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kube-rbac-proxy"
Oct 30 19:05:59.187448 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.187369    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="9578b5a7-d869-4370-8162-a6483fec8f2a" containerName="kserve-container"
Oct 30 19:05:59.190656 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.190633    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.241428 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.241401    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:05:59.289130 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.289089    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:05:59.289130 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.289120    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:05:59.289298 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.289159    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Oct 30 19:05:59.289298 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.289170    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:05:59.289298 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.289200    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-logger-predictor-serving-cert\""
Oct 30 19:05:59.289416 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.289404    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:05:59.290693 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.290661    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"]
Oct 30 19:05:59.299308 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.299287    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.299392 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.299321    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.299392 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.299341    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dnt\" (UniqueName: \"kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.299498 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.299481    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.299554 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.299541    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400180 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400152    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400301 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400202    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400301 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400232    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400301 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400255    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400301 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400274    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dnt\" (UniqueName: \"kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400604 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400564    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.400962 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400941    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.401036 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.400981    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.402755 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.402739    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.459529 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.459503    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dnt\" (UniqueName: \"kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt\") pod \"isvc-logger-predictor-55485cf944-t7wbh\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") " pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.509186 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.509165    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:05:59.926599 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:05:59.926503    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"]
Oct 30 19:05:59.929951 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:05:59.929924    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc66493_e938_4715_8206_f16be4ba2f96.slice/crio-9d461f3f29f693f949ee66d9295be8d1a0dc2bf965199eea07476d4cdc5381df WatchSource:0}: Error finding container 9d461f3f29f693f949ee66d9295be8d1a0dc2bf965199eea07476d4cdc5381df: Status 404 returned error can't find the container with id 9d461f3f29f693f949ee66d9295be8d1a0dc2bf965199eea07476d4cdc5381df
Oct 30 19:06:00.707502 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:00.707468    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerStarted","Data":"518bed442242d784a72de2e928f13030149f151e711a82126c33110cdaad1e26"}
Oct 30 19:06:00.707903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:00.707535    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerStarted","Data":"9d461f3f29f693f949ee66d9295be8d1a0dc2bf965199eea07476d4cdc5381df"}
Oct 30 19:06:04.722383 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:04.722349    5972 generic.go:356] "Generic (PLEG): container finished" podID="7cc66493-e938-4715-8206-f16be4ba2f96" containerID="518bed442242d784a72de2e928f13030149f151e711a82126c33110cdaad1e26" exitCode=0
Oct 30 19:06:04.722806 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:04.722425    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerDied","Data":"518bed442242d784a72de2e928f13030149f151e711a82126c33110cdaad1e26"}
Oct 30 19:06:05.728446 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.728414    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerStarted","Data":"fa4b06b4e32f9cd39c739e9ab798876ab4faf86ff6114128fbb647f29df224a3"}
Oct 30 19:06:05.728909 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.728479    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerStarted","Data":"80e59585f36fa053cece6316402f3a1dd5608076afe0d268b95730c6f30f5f65"}
Oct 30 19:06:05.728909 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.728519    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerStarted","Data":"c318e663fdd6b6c3d52d27216a48afa62b35271148fb995b71a2f07a298bf9ea"}
Oct 30 19:06:05.728909 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.728834    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:06:05.728909 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.728866    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:06:05.729064 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.728929    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:06:05.730204 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.730170    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:05.730973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.730953    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:05.846008 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:05.845961    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podStartSLOduration=3.031519014 podStartE2EDuration="6.845949448s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814430434s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:05.845104548 +0000 UTC m=+1615.637071930" watchObservedRunningTime="2026-10-30 19:06:05.845949448 +0000 UTC m=+1615.637916805"
Oct 30 19:06:06.733127 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:06.733088    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:06.733619 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:06.733560    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:11.737255 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:11.737229    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:06:11.737882 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:11.737846    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:11.738139 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:11.738111    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:21.738270 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:21.738186    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:21.738759 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:21.738714    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:22.870185 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:06:22.870147    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:06:31.738052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:31.738008    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:31.738517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:31.738463    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:41.738512 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:41.738463    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:41.739015 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:41.738974    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:51.737802 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:51.737764    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:06:51.738275 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:06:51.738251    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:52.870339 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:06:52.870303    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:07:01.737795 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:01.737752    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:07:01.738228 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:01.738186    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:07:11.738110 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:11.738072    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:07:11.738651 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:11.738566    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:07:21.738795 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:21.738766    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:07:21.739226 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:21.739007    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:07:22.869153 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:07:22.869121    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:07:35.152671 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:35.152639    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"]
Oct 30 19:07:35.153118 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:35.152872    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" containerID="cri-o://c318e663fdd6b6c3d52d27216a48afa62b35271148fb995b71a2f07a298bf9ea" gracePeriod=30
Oct 30 19:07:35.153118 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:35.152897    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" containerID="cri-o://fa4b06b4e32f9cd39c739e9ab798876ab4faf86ff6114128fbb647f29df224a3" gracePeriod=30
Oct 30 19:07:35.153118 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:35.152925    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" containerID="cri-o://80e59585f36fa053cece6316402f3a1dd5608076afe0d268b95730c6f30f5f65" gracePeriod=30
Oct 30 19:07:36.092236 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:36.092195    5972 generic.go:356] "Generic (PLEG): container finished" podID="7cc66493-e938-4715-8206-f16be4ba2f96" containerID="80e59585f36fa053cece6316402f3a1dd5608076afe0d268b95730c6f30f5f65" exitCode=0
Oct 30 19:07:36.092398 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:36.092271    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerDied","Data":"80e59585f36fa053cece6316402f3a1dd5608076afe0d268b95730c6f30f5f65"}
Oct 30 19:07:36.734219 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:36.734183    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 19:07:40.110489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:40.110403    5972 generic.go:356] "Generic (PLEG): container finished" podID="7cc66493-e938-4715-8206-f16be4ba2f96" containerID="c318e663fdd6b6c3d52d27216a48afa62b35271148fb995b71a2f07a298bf9ea" exitCode=0
Oct 30 19:07:40.110489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:40.110465    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerDied","Data":"c318e663fdd6b6c3d52d27216a48afa62b35271148fb995b71a2f07a298bf9ea"}
Oct 30 19:07:41.734025 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:41.733988    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 19:07:41.738305 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:41.738269    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:07:41.738645 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:41.738623    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:07:44.244652 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.244625    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"]
Oct 30 19:07:44.248173 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.248147    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.298903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.298875    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:07:44.299011 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.298952    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:07:44.346766 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.346730    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:07:44.346766 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.346730    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Oct 30 19:07:44.346766 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.346765    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:07:44.346982 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.346785    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:07:44.346982 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.346821    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:07:44.347803 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.347787    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"]
Oct 30 19:07:44.376863 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.376839    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.376972 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.376876    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcf8p\" (UniqueName: \"kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.376972 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.376896    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.376972 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.376917    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.377083 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.377055    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.477729 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.477701    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.477858 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.477741    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcf8p\" (UniqueName: \"kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.477858 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.477760    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.477858 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.477778    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.478019 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.477903    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.478158 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.478136    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.478427 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.478411    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.478470 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.478455    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.480248 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.480232    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.537045 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.536988    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcf8p\" (UniqueName: \"kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p\") pod \"isvc-sklearn-runtime-predictor-994454dbf-jr9pg\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.564910 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.564887    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:44.987687 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:44.987658    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"]
Oct 30 19:07:44.991568 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:07:44.991540    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b91285_378d_4c10_8c3e_ad8c000e5841.slice/crio-b793e4dce3caaf62e466b488e8ee6e825625aa9254a6c4f8729c787383e6271c WatchSource:0}: Error finding container b793e4dce3caaf62e466b488e8ee6e825625aa9254a6c4f8729c787383e6271c: Status 404 returned error can't find the container with id b793e4dce3caaf62e466b488e8ee6e825625aa9254a6c4f8729c787383e6271c
Oct 30 19:07:45.130329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:45.130296    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerStarted","Data":"74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421"}
Oct 30 19:07:45.130472 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:45.130361    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerStarted","Data":"b793e4dce3caaf62e466b488e8ee6e825625aa9254a6c4f8729c787383e6271c"}
Oct 30 19:07:46.734153 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:46.734114    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 19:07:46.734529 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:46.734241    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:07:51.733840 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:51.733807    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 19:07:51.738118 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:51.738089    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:07:51.738451 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:51.738428    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:07:52.159026 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:52.158950    5972 generic.go:356] "Generic (PLEG): container finished" podID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerID="74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421" exitCode=0
Oct 30 19:07:52.159026 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:52.159007    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerDied","Data":"74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421"}
Oct 30 19:07:52.870288 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:07:52.870252    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:07:53.165402 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:53.165328    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerStarted","Data":"d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20"}
Oct 30 19:07:53.165402 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:53.165393    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerStarted","Data":"c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44"}
Oct 30 19:07:53.165626 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:53.165610    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:53.287800 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:53.287756    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" podStartSLOduration=2.986149111 podStartE2EDuration="9.287743176s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.301594065s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:53.286877454 +0000 UTC m=+1723.078844741" watchObservedRunningTime="2026-10-30 19:07:53.287743176 +0000 UTC m=+1723.079710464"
Oct 30 19:07:54.169903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:54.169870    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:07:54.171004 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:54.170975    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Oct 30 19:07:55.174289 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:55.174254    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Oct 30 19:07:56.734014 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:07:56.733972    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 19:08:00.179271 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:00.179239    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:08:00.179902 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:00.179876    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Oct 30 19:08:01.733781 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:01.733737    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Oct 30 19:08:01.738181 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:01.738154    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Oct 30 19:08:01.738294 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:01.738286    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:08:01.738493 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:01.738477    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:08:01.738569 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:01.738557    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:08:05.241108 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.241078    5972 generic.go:356] "Generic (PLEG): container finished" podID="7cc66493-e938-4715-8206-f16be4ba2f96" containerID="fa4b06b4e32f9cd39c739e9ab798876ab4faf86ff6114128fbb647f29df224a3" exitCode=0
Oct 30 19:08:05.241420 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.241152    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerDied","Data":"fa4b06b4e32f9cd39c739e9ab798876ab4faf86ff6114128fbb647f29df224a3"}
Oct 30 19:08:05.307442 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.307415    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:08:05.472806 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.472784    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls\") pod \"7cc66493-e938-4715-8206-f16be4ba2f96\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") "
Oct 30 19:08:05.472921 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.472824    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert\") pod \"7cc66493-e938-4715-8206-f16be4ba2f96\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") "
Oct 30 19:08:05.472921 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.472869    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt\" (UniqueName: \"kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt\") pod \"7cc66493-e938-4715-8206-f16be4ba2f96\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") "
Oct 30 19:08:05.473000 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.472952    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config\") pod \"7cc66493-e938-4715-8206-f16be4ba2f96\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") "
Oct 30 19:08:05.473000 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.472981    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location\") pod \"7cc66493-e938-4715-8206-f16be4ba2f96\" (UID: \"7cc66493-e938-4715-8206-f16be4ba2f96\") "
Oct 30 19:08:05.473204 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.473179    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert" pod "7cc66493-e938-4715-8206-f16be4ba2f96" (UID: "7cc66493-e938-4715-8206-f16be4ba2f96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:05.473319 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.473298    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config" pod "7cc66493-e938-4715-8206-f16be4ba2f96" (UID: "7cc66493-e938-4715-8206-f16be4ba2f96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:05.473439 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.473414    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location" pod "7cc66493-e938-4715-8206-f16be4ba2f96" (UID: "7cc66493-e938-4715-8206-f16be4ba2f96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:05.474995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.474973    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls" pod "7cc66493-e938-4715-8206-f16be4ba2f96" (UID: "7cc66493-e938-4715-8206-f16be4ba2f96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:05.475172 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.475155    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt" pod "7cc66493-e938-4715-8206-f16be4ba2f96" (UID: "7cc66493-e938-4715-8206-f16be4ba2f96"). InnerVolumeSpecName "kube-api-access-z2dnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:05.573683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.573639    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.573683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.573667    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc66493-e938-4715-8206-f16be4ba2f96-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.573683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.573678    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc66493-e938-4715-8206-f16be4ba2f96-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.573683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.573689    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc66493-e938-4715-8206-f16be4ba2f96-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.573683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:05.573697    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dnt\" (UniqueName: \"kubernetes.io/projected/7cc66493-e938-4715-8206-f16be4ba2f96-kube-api-access-z2dnt\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:06.248258 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.248219    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh" event={"ID":"7cc66493-e938-4715-8206-f16be4ba2f96","Type":"ContainerDied","Data":"9d461f3f29f693f949ee66d9295be8d1a0dc2bf965199eea07476d4cdc5381df"}
Oct 30 19:08:06.248258 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.248241    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"
Oct 30 19:08:06.248809 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.248294    5972 scope.go:122] "RemoveContainer" containerID="fa4b06b4e32f9cd39c739e9ab798876ab4faf86ff6114128fbb647f29df224a3"
Oct 30 19:08:06.258751 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.258725    5972 scope.go:122] "RemoveContainer" containerID="80e59585f36fa053cece6316402f3a1dd5608076afe0d268b95730c6f30f5f65"
Oct 30 19:08:06.267315 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.267299    5972 scope.go:122] "RemoveContainer" containerID="c318e663fdd6b6c3d52d27216a48afa62b35271148fb995b71a2f07a298bf9ea"
Oct 30 19:08:06.275034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.275013    5972 scope.go:122] "RemoveContainer" containerID="518bed442242d784a72de2e928f13030149f151e711a82126c33110cdaad1e26"
Oct 30 19:08:06.456215 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.456188    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"]
Oct 30 19:08:06.503964 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:06.503891    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-logger-predictor-55485cf944-t7wbh"]
Oct 30 19:08:08.339003 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:08.338975    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" path="/var/lib/kubelet/pods/7cc66493-e938-4715-8206-f16be4ba2f96/volumes"
Oct 30 19:08:10.180644 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:10.180603    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:08:17.784290 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:17.784254    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"]
Oct 30 19:08:17.784827 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:17.784488    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container" containerID="cri-o://c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44" gracePeriod=30
Oct 30 19:08:17.784827 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:17.784539    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kube-rbac-proxy" containerID="cri-o://d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20" gracePeriod=30
Oct 30 19:08:18.301707 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.301677    5972 generic.go:356] "Generic (PLEG): container finished" podID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerID="d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20" exitCode=0
Oct 30 19:08:18.301875 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.301727    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerDied","Data":"d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20"}
Oct 30 19:08:18.921568 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.921544    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:08:18.988595 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988560    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert\") pod \"48b91285-378d-4c10-8c3e-ad8c000e5841\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") "
Oct 30 19:08:18.988736 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988618    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config\") pod \"48b91285-378d-4c10-8c3e-ad8c000e5841\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") "
Oct 30 19:08:18.988736 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988666    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls\" (UniqueName: \"kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls\") pod \"48b91285-378d-4c10-8c3e-ad8c000e5841\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") "
Oct 30 19:08:18.988736 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988719    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location\") pod \"48b91285-378d-4c10-8c3e-ad8c000e5841\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") "
Oct 30 19:08:18.988908 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988747    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p\" (UniqueName: \"kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p\") pod \"48b91285-378d-4c10-8c3e-ad8c000e5841\" (UID: \"48b91285-378d-4c10-8c3e-ad8c000e5841\") "
Oct 30 19:08:18.988962 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988904    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config" pod "48b91285-378d-4c10-8c3e-ad8c000e5841" (UID: "48b91285-378d-4c10-8c3e-ad8c000e5841"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:18.988962 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.988924    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert" pod "48b91285-378d-4c10-8c3e-ad8c000e5841" (UID: "48b91285-378d-4c10-8c3e-ad8c000e5841"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:18.989144 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.989124    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:18.989228 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.989150    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:18.990990 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.990951    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls" pod "48b91285-378d-4c10-8c3e-ad8c000e5841" (UID: "48b91285-378d-4c10-8c3e-ad8c000e5841"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:18.990990 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:18.990984    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p" pod "48b91285-378d-4c10-8c3e-ad8c000e5841" (UID: "48b91285-378d-4c10-8c3e-ad8c000e5841"). InnerVolumeSpecName "kube-api-access-xcf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:19.012540 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.012488    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location" pod "48b91285-378d-4c10-8c3e-ad8c000e5841" (UID: "48b91285-378d-4c10-8c3e-ad8c000e5841"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:19.090570 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.090537    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcf8p\" (UniqueName: \"kubernetes.io/projected/48b91285-378d-4c10-8c3e-ad8c000e5841-kube-api-access-xcf8p\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:19.090570 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.090569    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48b91285-378d-4c10-8c3e-ad8c000e5841-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:19.090785 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.090597    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48b91285-378d-4c10-8c3e-ad8c000e5841-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:19.308020 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.307932    5972 generic.go:356] "Generic (PLEG): container finished" podID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerID="c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44" exitCode=0
Oct 30 19:08:19.308020 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.307986    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerDied","Data":"c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44"}
Oct 30 19:08:19.308020 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.307999    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"
Oct 30 19:08:19.308287 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.308043    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg" event={"ID":"48b91285-378d-4c10-8c3e-ad8c000e5841","Type":"ContainerDied","Data":"b793e4dce3caaf62e466b488e8ee6e825625aa9254a6c4f8729c787383e6271c"}
Oct 30 19:08:19.308287 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.308072    5972 scope.go:122] "RemoveContainer" containerID="d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20"
Oct 30 19:08:19.317259 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.317244    5972 scope.go:122] "RemoveContainer" containerID="c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44"
Oct 30 19:08:19.324414 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.324399    5972 scope.go:122] "RemoveContainer" containerID="74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421"
Oct 30 19:08:19.331272 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.331257    5972 scope.go:122] "RemoveContainer" containerID="d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20"
Oct 30 19:08:19.331539 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:08:19.331520    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20\": container with ID starting with d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20 not found: ID does not exist" containerID="d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20"
Oct 30 19:08:19.331608 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.331549    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20"} err="failed to get container status \"d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20\": rpc error: code = NotFound desc = could not find container \"d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20\": container with ID starting with d4fe7b10f1ea80531f727bb83b3d88ed4f9452e15e9a66b05fcb768bc8d80b20 not found: ID does not exist"
Oct 30 19:08:19.331608 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.331569    5972 scope.go:122] "RemoveContainer" containerID="c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44"
Oct 30 19:08:19.331811 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:08:19.331795    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44\": container with ID starting with c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44 not found: ID does not exist" containerID="c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44"
Oct 30 19:08:19.331852 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.331816    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44"} err="failed to get container status \"c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44\": rpc error: code = NotFound desc = could not find container \"c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44\": container with ID starting with c6c18916afc0395a1c453ce5ba3945182e31d896bab128ddf8c9a642995a7e44 not found: ID does not exist"
Oct 30 19:08:19.331852 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.331829    5972 scope.go:122] "RemoveContainer" containerID="74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421"
Oct 30 19:08:19.332042 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:08:19.332025    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421\": container with ID starting with 74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421 not found: ID does not exist" containerID="74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421"
Oct 30 19:08:19.332098 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.332051    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421"} err="failed to get container status \"74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421\": rpc error: code = NotFound desc = could not find container \"74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421\": container with ID starting with 74980c92719afee42928c129d3f488fcdc9374d36df28853e79cb48151d78421 not found: ID does not exist"
Oct 30 19:08:19.466755 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.466726    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"]
Oct 30 19:08:19.514477 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:19.514450    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-runtime-predictor-994454dbf-jr9pg"]
Oct 30 19:08:20.338259 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:08:20.338229    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" path="/var/lib/kubelet/pods/48b91285-378d-4c10-8c3e-ad8c000e5841/volumes"
Oct 30 19:08:22.869667 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:08:22.869632    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:08:52.870043 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:08:52.870005    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:09:10.527495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:10.527458    5972 scope.go:122] "RemoveContainer" containerID="7d3a84d3975f9291c88f5da0ec89275b46590f220f6c111c331948896d51849c"
Oct 30 19:09:10.535743 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:10.535722    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:09:10.537440 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:10.537414    5972 scope.go:122] "RemoveContainer" containerID="fa5ea902aec4bb1b5a60336dfae3f7154cd87d26a97c920787ea7ecc71ae898d"
Oct 30 19:09:10.548998 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:10.548982    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:09:22.869000 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:09:22.868927    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:09:34.025850 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.025815    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"]
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026124    5972 cpu_manager.go:414] "containerMap: removing container" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026135    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026146    5972 cpu_manager.go:414] "containerMap: removing container" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kube-rbac-proxy"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026151    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kube-rbac-proxy"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026169    5972 cpu_manager.go:414] "containerMap: removing container" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026175    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026185    5972 cpu_manager.go:414] "containerMap: removing container" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="storage-initializer"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026190    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="storage-initializer"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026202    5972 cpu_manager.go:414] "containerMap: removing container" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026207    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026213    5972 cpu_manager.go:414] "containerMap: removing container" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="storage-initializer"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026217    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="storage-initializer"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026223    5972 cpu_manager.go:414] "containerMap: removing container" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container"
Oct 30 19:09:34.026246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026227    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container"
Oct 30 19:09:34.026680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026276    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kube-rbac-proxy"
Oct 30 19:09:34.026680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026283    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kserve-container"
Oct 30 19:09:34.026680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026290    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="agent"
Oct 30 19:09:34.026680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026295    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b91285-378d-4c10-8c3e-ad8c000e5841" containerName="kube-rbac-proxy"
Oct 30 19:09:34.026680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.026303    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc66493-e938-4715-8206-f16be4ba2f96" containerName="kserve-container"
Oct 30 19:09:34.029316 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.029294    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.081143 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.081116    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:09:34.081297 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.081178    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:09:34.095534 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.095510    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr498\" (UniqueName: \"kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.095663 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.095546    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.095730 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.095675    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.095730 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.095712    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.095796 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.095780    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.129119 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.129084    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:09:34.129259 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.129145    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:09:34.129460 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.129428    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:09:34.129460 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.129460    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:09:34.129668 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.129464    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Oct 30 19:09:34.130271 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.130250    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"]
Oct 30 19:09:34.196851 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.196831    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.196955 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.196898    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr498\" (UniqueName: \"kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.196955 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.196921    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.197046 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.196960    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.197238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.197213    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.197370 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.197259    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.197577 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.197554    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.197721 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.197700    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.199923 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.199899    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.257009 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.256983    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr498\" (UniqueName: \"kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498\") pod \"isvc-sklearn-v2-predictor-7f446df4d7-4nzmq\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.345960 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.345904    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:34.759606 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:34.759568    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"]
Oct 30 19:09:34.762451 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:09:34.762409    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542193c8_4f19_4d4a_8298_ef518dbcdce4.slice/crio-9776762d0da33f7cf372b905c2f2c927d8ddf49e36d8fa43843ac76d96220eed WatchSource:0}: Error finding container 9776762d0da33f7cf372b905c2f2c927d8ddf49e36d8fa43843ac76d96220eed: Status 404 returned error can't find the container with id 9776762d0da33f7cf372b905c2f2c927d8ddf49e36d8fa43843ac76d96220eed
Oct 30 19:09:35.599928 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:35.599895    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerStarted","Data":"304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb"}
Oct 30 19:09:35.600329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:35.599959    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerStarted","Data":"9776762d0da33f7cf372b905c2f2c927d8ddf49e36d8fa43843ac76d96220eed"}
Oct 30 19:09:38.611953 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:38.611920    5972 generic.go:356] "Generic (PLEG): container finished" podID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerID="304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb" exitCode=0
Oct 30 19:09:38.612278 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:38.611992    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerDied","Data":"304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb"}
Oct 30 19:09:39.617781 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:39.617750    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerStarted","Data":"fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d"}
Oct 30 19:09:39.618161 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:39.617808    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerStarted","Data":"a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46"}
Oct 30 19:09:39.618161 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:39.617994    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:39.735833 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:39.735782    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podStartSLOduration=3.033258087 podStartE2EDuration="6.735769773s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.702511686s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:39.735380986 +0000 UTC m=+1829.527348298" watchObservedRunningTime="2026-10-30 19:09:39.735769773 +0000 UTC m=+1829.527737060"
Oct 30 19:09:40.621168 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:40.621140    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:40.622400 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:40.622374    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:09:41.625831 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:41.625796    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:09:46.630915 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:46.630842    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:09:46.631386 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:46.631361    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:09:52.869464 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:09:52.869417    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:09:56.631535 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:09:56.631490    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:10:06.632128 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:10:06.632083    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:10:16.632087 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:10:16.632045    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:10:22.869742 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:10:22.869703    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:10:26.632249 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:10:26.632215    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:10:36.631557 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:10:36.631521    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:10:46.632129 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:10:46.632092    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 19:10:52.869735 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:10:52.869655    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:10:53.334697 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:10:53.334671    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:11:05.259068 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:05.259032    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"]
Oct 30 19:11:05.259562 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:05.259448    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container" containerID="cri-o://a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46" gracePeriod=30
Oct 30 19:11:05.259562 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:05.259480    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kube-rbac-proxy" containerID="cri-o://fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d" gracePeriod=30
Oct 30 19:11:05.950258 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:05.950225    5972 generic.go:356] "Generic (PLEG): container finished" podID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerID="fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d" exitCode=0
Oct 30 19:11:05.950419 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:05.950272    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerDied","Data":"fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d"}
Oct 30 19:11:06.626981 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:06.626943    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Oct 30 19:11:09.900465 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.900445    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:11:09.967544 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.967487    5972 generic.go:356] "Generic (PLEG): container finished" podID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerID="a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46" exitCode=0
Oct 30 19:11:09.967673 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.967549    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"
Oct 30 19:11:09.967673 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.967575    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerDied","Data":"a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46"}
Oct 30 19:11:09.967673 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.967651    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq" event={"ID":"542193c8-4f19-4d4a-8298-ef518dbcdce4","Type":"ContainerDied","Data":"9776762d0da33f7cf372b905c2f2c927d8ddf49e36d8fa43843ac76d96220eed"}
Oct 30 19:11:09.967775 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.967679    5972 scope.go:122] "RemoveContainer" containerID="fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d"
Oct 30 19:11:09.975329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.975311    5972 scope.go:122] "RemoveContainer" containerID="a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46"
Oct 30 19:11:09.982828 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.982808    5972 scope.go:122] "RemoveContainer" containerID="304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb"
Oct 30 19:11:09.989398 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989374    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config\") pod \"542193c8-4f19-4d4a-8298-ef518dbcdce4\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") "
Oct 30 19:11:09.989501 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989430    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls\") pod \"542193c8-4f19-4d4a-8298-ef518dbcdce4\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") "
Oct 30 19:11:09.989501 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989480    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert\") pod \"542193c8-4f19-4d4a-8298-ef518dbcdce4\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") "
Oct 30 19:11:09.989633 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989513    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498\" (UniqueName: \"kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498\") pod \"542193c8-4f19-4d4a-8298-ef518dbcdce4\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") "
Oct 30 19:11:09.989698 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989634    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location\") pod \"542193c8-4f19-4d4a-8298-ef518dbcdce4\" (UID: \"542193c8-4f19-4d4a-8298-ef518dbcdce4\") "
Oct 30 19:11:09.989838 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989814    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config" pod "542193c8-4f19-4d4a-8298-ef518dbcdce4" (UID: "542193c8-4f19-4d4a-8298-ef518dbcdce4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:09.989838 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989852    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert" pod "542193c8-4f19-4d4a-8298-ef518dbcdce4" (UID: "542193c8-4f19-4d4a-8298-ef518dbcdce4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:09.990052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.989881    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:09.990117 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.990069    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location" pod "542193c8-4f19-4d4a-8298-ef518dbcdce4" (UID: "542193c8-4f19-4d4a-8298-ef518dbcdce4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:09.990788 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.990772    5972 scope.go:122] "RemoveContainer" containerID="fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d"
Oct 30 19:11:09.991087 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:11:09.991066    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d\": container with ID starting with fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d not found: ID does not exist" containerID="fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d"
Oct 30 19:11:09.991167 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.991099    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d"} err="failed to get container status \"fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d\": rpc error: code = NotFound desc = could not find container \"fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d\": container with ID starting with fc406ff52a3b726ac03308b7be10f696fe305731362429908f116fc6f39e025d not found: ID does not exist"
Oct 30 19:11:09.991167 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.991124    5972 scope.go:122] "RemoveContainer" containerID="a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46"
Oct 30 19:11:09.991392 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:11:09.991377    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46\": container with ID starting with a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46 not found: ID does not exist" containerID="a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46"
Oct 30 19:11:09.991468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.991412    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46"} err="failed to get container status \"a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46\": rpc error: code = NotFound desc = could not find container \"a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46\": container with ID starting with a97a200c631769ddb9da2f5598ed7387677d073ce2e39333856301acb1844b46 not found: ID does not exist"
Oct 30 19:11:09.991468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.991439    5972 scope.go:122] "RemoveContainer" containerID="304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb"
Oct 30 19:11:09.991762 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.991742    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls" pod "542193c8-4f19-4d4a-8298-ef518dbcdce4" (UID: "542193c8-4f19-4d4a-8298-ef518dbcdce4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:09.991831 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:11:09.991761    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb\": container with ID starting with 304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb not found: ID does not exist" containerID="304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb"
Oct 30 19:11:09.991831 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.991784    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb"} err="failed to get container status \"304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb\": rpc error: code = NotFound desc = could not find container \"304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb\": container with ID starting with 304de8b56c000c2c355c065a633712b3588cf2490f954fb9971e2294041b43cb not found: ID does not exist"
Oct 30 19:11:09.992159 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:09.992145    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498" pod "542193c8-4f19-4d4a-8298-ef518dbcdce4" (UID: "542193c8-4f19-4d4a-8298-ef518dbcdce4"). InnerVolumeSpecName "kube-api-access-qr498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:10.090422 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:10.090395    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542193c8-4f19-4d4a-8298-ef518dbcdce4-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:10.090422 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:10.090419    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542193c8-4f19-4d4a-8298-ef518dbcdce4-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:10.090422 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:10.090430    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/542193c8-4f19-4d4a-8298-ef518dbcdce4-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:10.090422 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:10.090438    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr498\" (UniqueName: \"kubernetes.io/projected/542193c8-4f19-4d4a-8298-ef518dbcdce4-kube-api-access-qr498\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:10.383142 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:10.383107    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"]
Oct 30 19:11:10.387825 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:10.387805    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-predictor-7f446df4d7-4nzmq"]
Oct 30 19:11:12.338291 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:12.338257    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" path="/var/lib/kubelet/pods/542193c8-4f19-4d4a-8298-ef518dbcdce4/volumes"
Oct 30 19:11:14.339825 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.339794    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"]
Oct 30 19:11:14.340218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340114    5972 cpu_manager.go:414] "containerMap: removing container" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container"
Oct 30 19:11:14.340218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340126    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container"
Oct 30 19:11:14.340218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340155    5972 cpu_manager.go:414] "containerMap: removing container" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kube-rbac-proxy"
Oct 30 19:11:14.340218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340161    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kube-rbac-proxy"
Oct 30 19:11:14.340218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340170    5972 cpu_manager.go:414] "containerMap: removing container" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="storage-initializer"
Oct 30 19:11:14.340218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340174    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="storage-initializer"
Oct 30 19:11:14.340399 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340230    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kube-rbac-proxy"
Oct 30 19:11:14.340399 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.340241    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="542193c8-4f19-4d4a-8298-ef518dbcdce4" containerName="kserve-container"
Oct 30 19:11:14.344963 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.344948    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.396715 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.396694    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:11:14.426497 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.426472    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.426615 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.426509    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9hd\" (UniqueName: \"kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.426615 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.426534    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.426705 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.426614    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.426741 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.426706    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.444513 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.444478    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:11:14.444513 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.444492    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:11:14.444684 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.444514    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:11:14.444684 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.444544    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:11:14.444684 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.444545    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Oct 30 19:11:14.444684 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.444660    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Oct 30 19:11:14.445489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.445472    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"]
Oct 30 19:11:14.527369 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.527342    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.527496 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.527383    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9hd\" (UniqueName: \"kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.527496 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.527416    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.527496 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.527442    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.527496 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.527491    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.528034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.528011    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.528127 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.528111    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.528205 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.528187    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.530191 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.530174    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.587158 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.587127    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9hd\" (UniqueName: \"kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd\") pod \"isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:14.660476 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:14.660421    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:15.074380 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:15.074150    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"]
Oct 30 19:11:15.076978 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:11:15.076948    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073e80ea_57e1_4df9_b531_d526daa2fc4b.slice/crio-f3026919d841ae9a36944adae07a8c2ded20f757daf4da5d1bdf4e6e3884a1db WatchSource:0}: Error finding container f3026919d841ae9a36944adae07a8c2ded20f757daf4da5d1bdf4e6e3884a1db: Status 404 returned error can't find the container with id f3026919d841ae9a36944adae07a8c2ded20f757daf4da5d1bdf4e6e3884a1db
Oct 30 19:11:15.993716 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:15.993682    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerStarted","Data":"8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204"}
Oct 30 19:11:15.994093 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:15.993738    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerStarted","Data":"f3026919d841ae9a36944adae07a8c2ded20f757daf4da5d1bdf4e6e3884a1db"}
Oct 30 19:11:19.006168 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:19.006140    5972 generic.go:356] "Generic (PLEG): container finished" podID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerID="8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204" exitCode=0
Oct 30 19:11:19.006471 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:19.006209    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerDied","Data":"8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204"}
Oct 30 19:11:20.012991 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:20.012953    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerStarted","Data":"ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125"}
Oct 30 19:11:20.013369 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:20.013020    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerStarted","Data":"ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a"}
Oct 30 19:11:20.013369 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:20.013211    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:20.129442 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:20.129392    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podStartSLOduration=2.2852543819999998 podStartE2EDuration="6.129379622s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.84412524s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:20.128741821 +0000 UTC m=+1929.920709109" watchObservedRunningTime="2026-10-30 19:11:20.129379622 +0000 UTC m=+1929.921346910"
Oct 30 19:11:21.016040 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:21.016009    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:21.017248 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:21.017221    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:11:22.020603 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:22.020553    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:11:22.870245 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:11:22.870206    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:11:27.025097 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:27.025073    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:11:27.025522 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:27.025499    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:11:37.026102 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:37.026066    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:11:47.026030 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:47.025991    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:11:52.869798 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:11:52.869761    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:11:57.026103 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:11:57.026064    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:12:07.025568 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:07.025532    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:12:17.025950 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:17.025910    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:12:22.870389 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:12:22.870306    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:12:27.025671 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:27.025637    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:12:37.026760 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:37.026729    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:12:44.915782 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:44.915747    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"]
Oct 30 19:12:44.916332 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:44.916038    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" containerID="cri-o://ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a" gracePeriod=30
Oct 30 19:12:44.916332 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:44.916073    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kube-rbac-proxy" containerID="cri-o://ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125" gracePeriod=30
Oct 30 19:12:45.352699 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:45.352668    5972 generic.go:356] "Generic (PLEG): container finished" podID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerID="ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125" exitCode=0
Oct 30 19:12:45.352865 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:45.352741    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerDied","Data":"ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125"}
Oct 30 19:12:47.021705 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:47.021670    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Oct 30 19:12:47.025957 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:47.025934    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 19:12:49.757372 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.757349    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:12:49.820421 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820399    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls\") pod \"073e80ea-57e1-4df9-b531-d526daa2fc4b\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") "
Oct 30 19:12:49.820545 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820439    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd\" (UniqueName: \"kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd\") pod \"073e80ea-57e1-4df9-b531-d526daa2fc4b\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") "
Oct 30 19:12:49.820545 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820461    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config\") pod \"073e80ea-57e1-4df9-b531-d526daa2fc4b\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") "
Oct 30 19:12:49.820545 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820485    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location\") pod \"073e80ea-57e1-4df9-b531-d526daa2fc4b\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") "
Oct 30 19:12:49.820545 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820532    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert\") pod \"073e80ea-57e1-4df9-b531-d526daa2fc4b\" (UID: \"073e80ea-57e1-4df9-b531-d526daa2fc4b\") "
Oct 30 19:12:49.820885 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820841    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location" pod "073e80ea-57e1-4df9-b531-d526daa2fc4b" (UID: "073e80ea-57e1-4df9-b531-d526daa2fc4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:49.820885 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820858    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config" pod "073e80ea-57e1-4df9-b531-d526daa2fc4b" (UID: "073e80ea-57e1-4df9-b531-d526daa2fc4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:49.821068 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.820943    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert" pod "073e80ea-57e1-4df9-b531-d526daa2fc4b" (UID: "073e80ea-57e1-4df9-b531-d526daa2fc4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:49.822695 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.822668    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd" pod "073e80ea-57e1-4df9-b531-d526daa2fc4b" (UID: "073e80ea-57e1-4df9-b531-d526daa2fc4b"). InnerVolumeSpecName "kube-api-access-2m9hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:49.822780 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.822705    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls" pod "073e80ea-57e1-4df9-b531-d526daa2fc4b" (UID: "073e80ea-57e1-4df9-b531-d526daa2fc4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:49.922060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.921996    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9hd\" (UniqueName: \"kubernetes.io/projected/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-api-access-2m9hd\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:49.922060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.922024    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:49.922060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.922035    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073e80ea-57e1-4df9-b531-d526daa2fc4b-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:49.922060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.922045    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073e80ea-57e1-4df9-b531-d526daa2fc4b-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:49.922060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:49.922054    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073e80ea-57e1-4df9-b531-d526daa2fc4b-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:50.375215 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.375191    5972 generic.go:356] "Generic (PLEG): container finished" podID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerID="ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a" exitCode=0
Oct 30 19:12:50.375409 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.375245    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"
Oct 30 19:12:50.375409 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.375248    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerDied","Data":"ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a"}
Oct 30 19:12:50.375409 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.375390    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n" event={"ID":"073e80ea-57e1-4df9-b531-d526daa2fc4b","Type":"ContainerDied","Data":"f3026919d841ae9a36944adae07a8c2ded20f757daf4da5d1bdf4e6e3884a1db"}
Oct 30 19:12:50.375532 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.375433    5972 scope.go:122] "RemoveContainer" containerID="ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125"
Oct 30 19:12:50.383508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.383489    5972 scope.go:122] "RemoveContainer" containerID="ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a"
Oct 30 19:12:50.390541 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.390528    5972 scope.go:122] "RemoveContainer" containerID="8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204"
Oct 30 19:12:50.399339 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.399319    5972 scope.go:122] "RemoveContainer" containerID="ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125"
Oct 30 19:12:50.399626 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:12:50.399603    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125\": container with ID starting with ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125 not found: ID does not exist" containerID="ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125"
Oct 30 19:12:50.399673 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.399637    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125"} err="failed to get container status \"ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125\": rpc error: code = NotFound desc = could not find container \"ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125\": container with ID starting with ede8321be11d9f65f5be7c58b2313a9062e65244d18d830503e9e8433213b125 not found: ID does not exist"
Oct 30 19:12:50.399673 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.399655    5972 scope.go:122] "RemoveContainer" containerID="ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a"
Oct 30 19:12:50.399881 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:12:50.399867    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a\": container with ID starting with ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a not found: ID does not exist" containerID="ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a"
Oct 30 19:12:50.399920 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.399886    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a"} err="failed to get container status \"ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a\": rpc error: code = NotFound desc = could not find container \"ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a\": container with ID starting with ce30701d64c30c886270ce0ec668e90eeddf8b45f0c6bf859ca1bcc1556be89a not found: ID does not exist"
Oct 30 19:12:50.399920 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.399899    5972 scope.go:122] "RemoveContainer" containerID="8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204"
Oct 30 19:12:50.400088 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:12:50.400072    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204\": container with ID starting with 8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204 not found: ID does not exist" containerID="8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204"
Oct 30 19:12:50.400135 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.400095    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204"} err="failed to get container status \"8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204\": rpc error: code = NotFound desc = could not find container \"8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204\": container with ID starting with 8dc0a6bdce8a7486a63caa0e5ac01ca43c1ca0433b132aa9213470adfe707204 not found: ID does not exist"
Oct 30 19:12:50.536978 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.536950    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"]
Oct 30 19:12:50.584518 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:50.584493    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-mixed-predictor-6b789bbf54-24l6n"]
Oct 30 19:12:52.338144 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:52.338111    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" path="/var/lib/kubelet/pods/073e80ea-57e1-4df9-b531-d526daa2fc4b/volumes"
Oct 30 19:12:52.869573 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:12:52.869535    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:12:54.490430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490402    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"]
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490724    5972 cpu_manager.go:414] "containerMap: removing container" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kube-rbac-proxy"
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490736    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kube-rbac-proxy"
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490771    5972 cpu_manager.go:414] "containerMap: removing container" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="storage-initializer"
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490775    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="storage-initializer"
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490783    5972 cpu_manager.go:414] "containerMap: removing container" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container"
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490787    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container"
Oct 30 19:12:54.490836 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490835    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kserve-container"
Oct 30 19:12:54.491049 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.490843    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="073e80ea-57e1-4df9-b531-d526daa2fc4b" containerName="kube-rbac-proxy"
Oct 30 19:12:54.495421 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.495403    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.546018 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.545993    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Oct 30 19:12:54.559844 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.559821    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.559960 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.559883    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.559960 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.559924    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmmj\" (UniqueName: \"kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.560076 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.559973    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.560076 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.560011    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.593876 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.593791    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:12:54.593876 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.593791    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-tensorflow-predictor-serving-cert\""
Oct 30 19:12:54.593876 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.593823    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:12:54.593876 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.593862    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:12:54.594145 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.593916    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:12:54.594145 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.593927    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:12:54.594866 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.594844    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"]
Oct 30 19:12:54.661074 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661049    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661232 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661077    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmmj\" (UniqueName: \"kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661232 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661095    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661232 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661115    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661232 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661150    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661624 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661604    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661844 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661829    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.661893 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.661835    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.663687 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.663671    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.720892 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.720866    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmmj\" (UniqueName: \"kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj\") pod \"isvc-tensorflow-predictor-66f96c784-xd52x\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:54.811973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:54.811895    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:12:55.225496 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:55.225471    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"]
Oct 30 19:12:55.228611 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:12:55.228566    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267bc2da_c737_4d6a_90d6_71a2fbd49d18.slice/crio-34a04a280029e956f74159bdd563748e8aef71a7fd796fd20ad75dd20592e4ee WatchSource:0}: Error finding container 34a04a280029e956f74159bdd563748e8aef71a7fd796fd20ad75dd20592e4ee: Status 404 returned error can't find the container with id 34a04a280029e956f74159bdd563748e8aef71a7fd796fd20ad75dd20592e4ee
Oct 30 19:12:55.395272 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:55.395238    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerStarted","Data":"b0078ada86dfced9b717de80af70ec4a980a84ce7a2643216bd59b3a564a5218"}
Oct 30 19:12:55.395434 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:12:55.395297    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerStarted","Data":"34a04a280029e956f74159bdd563748e8aef71a7fd796fd20ad75dd20592e4ee"}
Oct 30 19:13:01.418451 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:01.418415    5972 generic.go:356] "Generic (PLEG): container finished" podID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerID="b0078ada86dfced9b717de80af70ec4a980a84ce7a2643216bd59b3a564a5218" exitCode=0
Oct 30 19:13:01.418946 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:01.418514    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerDied","Data":"b0078ada86dfced9b717de80af70ec4a980a84ce7a2643216bd59b3a564a5218"}
Oct 30 19:13:01.419907 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:01.419888    5972 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:13:06.442842 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:06.442811    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerStarted","Data":"2b9029790e27835c341098d75aaf9ad4cbd6547e5475fb902f3a0831a10365d5"}
Oct 30 19:13:06.443266 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:06.442870    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerStarted","Data":"4a993175c674c591fe8380d5f2ee2fcfe33e1fec3375ff109d2f4797c51f4754"}
Oct 30 19:13:06.443266 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:06.443061    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:13:06.585678 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:06.585630    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podStartSLOduration=2.705044637 podStartE2EDuration="12.585616377s" totalImagesPullingTime="4.73249873s" totalInitContainerRuntime="5.14807301s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:06.58460596 +0000 UTC m=+2036.376573248" watchObservedRunningTime="2026-10-30 19:13:06.585616377 +0000 UTC m=+2036.377583740"
Oct 30 19:13:07.446982 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:07.446944    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:13:07.448144 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:07.448113    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 19:13:08.450181 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:08.450134    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 19:13:13.454470 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:13.454442    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:13:13.455013 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:13.454986    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 19:13:22.870043 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:13:22.870009    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:13:23.455059 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:23.455031    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:13:44.377114 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:44.377081    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"]
Oct 30 19:13:44.377566 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:44.377318    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container" containerID="cri-o://4a993175c674c591fe8380d5f2ee2fcfe33e1fec3375ff109d2f4797c51f4754" gracePeriod=30
Oct 30 19:13:44.377566 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:44.377342    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" containerID="cri-o://2b9029790e27835c341098d75aaf9ad4cbd6547e5475fb902f3a0831a10365d5" gracePeriod=30
Oct 30 19:13:44.591717 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:44.591690    5972 generic.go:356] "Generic (PLEG): container finished" podID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerID="2b9029790e27835c341098d75aaf9ad4cbd6547e5475fb902f3a0831a10365d5" exitCode=0
Oct 30 19:13:44.591862 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:44.591768    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerDied","Data":"2b9029790e27835c341098d75aaf9ad4cbd6547e5475fb902f3a0831a10365d5"}
Oct 30 19:13:48.450408 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:48.450333    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 19:13:52.869482 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:13:52.869443    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:13:53.450553 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:53.450514    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 19:13:58.451243 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:58.451197    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 19:13:58.451757 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:13:58.451384    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:14:03.450317 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:03.450272    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 19:14:08.451201 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:08.451158    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 19:14:10.563402 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:10.563375    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:14:10.583314 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:10.583291    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:14:13.451161 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:13.451119    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 19:14:14.710029 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:14.710000    5972 generic.go:356] "Generic (PLEG): container finished" podID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerID="4a993175c674c591fe8380d5f2ee2fcfe33e1fec3375ff109d2f4797c51f4754" exitCode=137
Oct 30 19:14:14.710400 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:14.710076    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerDied","Data":"4a993175c674c591fe8380d5f2ee2fcfe33e1fec3375ff109d2f4797c51f4754"}
Oct 30 19:14:15.013568 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.013545    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:14:15.117369 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117342    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config\") pod \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") "
Oct 30 19:14:15.117495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117395    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls\" (UniqueName: \"kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls\") pod \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") "
Oct 30 19:14:15.117495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117423    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj\" (UniqueName: \"kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj\") pod \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") "
Oct 30 19:14:15.117495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117484    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location\") pod \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") "
Oct 30 19:14:15.117655 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117512    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert\") pod \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\" (UID: \"267bc2da-c737-4d6a-90d6-71a2fbd49d18\") "
Oct 30 19:14:15.117728 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117704    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config" pod "267bc2da-c737-4d6a-90d6-71a2fbd49d18" (UID: "267bc2da-c737-4d6a-90d6-71a2fbd49d18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:15.117917 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117893    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.118049 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.117929    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert" pod "267bc2da-c737-4d6a-90d6-71a2fbd49d18" (UID: "267bc2da-c737-4d6a-90d6-71a2fbd49d18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:15.119683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.119664    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls" pod "267bc2da-c737-4d6a-90d6-71a2fbd49d18" (UID: "267bc2da-c737-4d6a-90d6-71a2fbd49d18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:15.119777 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.119748    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj" pod "267bc2da-c737-4d6a-90d6-71a2fbd49d18" (UID: "267bc2da-c737-4d6a-90d6-71a2fbd49d18"). InnerVolumeSpecName "kube-api-access-8xmmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:15.127933 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.127896    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location" pod "267bc2da-c737-4d6a-90d6-71a2fbd49d18" (UID: "267bc2da-c737-4d6a-90d6-71a2fbd49d18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:15.219084 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.219055    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/267bc2da-c737-4d6a-90d6-71a2fbd49d18-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.219084 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.219081    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmmj\" (UniqueName: \"kubernetes.io/projected/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kube-api-access-8xmmj\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.219238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.219091    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/267bc2da-c737-4d6a-90d6-71a2fbd49d18-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.219238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.219101    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/267bc2da-c737-4d6a-90d6-71a2fbd49d18-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.716149 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.716124    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"
Oct 30 19:14:15.716569 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.716145    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x" event={"ID":"267bc2da-c737-4d6a-90d6-71a2fbd49d18","Type":"ContainerDied","Data":"34a04a280029e956f74159bdd563748e8aef71a7fd796fd20ad75dd20592e4ee"}
Oct 30 19:14:15.716569 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.716218    5972 scope.go:122] "RemoveContainer" containerID="2b9029790e27835c341098d75aaf9ad4cbd6547e5475fb902f3a0831a10365d5"
Oct 30 19:14:15.724742 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.724726    5972 scope.go:122] "RemoveContainer" containerID="4a993175c674c591fe8380d5f2ee2fcfe33e1fec3375ff109d2f4797c51f4754"
Oct 30 19:14:15.732445 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.732430    5972 scope.go:122] "RemoveContainer" containerID="b0078ada86dfced9b717de80af70ec4a980a84ce7a2643216bd59b3a564a5218"
Oct 30 19:14:15.875485 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.875461    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"]
Oct 30 19:14:15.923385 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:15.923358    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-predictor-66f96c784-xd52x"]
Oct 30 19:14:16.338380 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:16.338349    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" path="/var/lib/kubelet/pods/267bc2da-c737-4d6a-90d6-71a2fbd49d18/volumes"
Oct 30 19:14:19.836597 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836562    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"]
Oct 30 19:14:19.836985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836894    5972 cpu_manager.go:414] "containerMap: removing container" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="storage-initializer"
Oct 30 19:14:19.836985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836907    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="storage-initializer"
Oct 30 19:14:19.836985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836932    5972 cpu_manager.go:414] "containerMap: removing container" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy"
Oct 30 19:14:19.836985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836937    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy"
Oct 30 19:14:19.836985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836951    5972 cpu_manager.go:414] "containerMap: removing container" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container"
Oct 30 19:14:19.836985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.836957    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container"
Oct 30 19:14:19.837205 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.837003    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kserve-container"
Oct 30 19:14:19.837205 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.837015    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="267bc2da-c737-4d6a-90d6-71a2fbd49d18" containerName="kube-rbac-proxy"
Oct 30 19:14:19.841658 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.841638    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.854116 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.854096    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.854221 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.854131    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.854221 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.854156    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lks6w\" (UniqueName: \"kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.854291 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.854250    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.854329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.854293    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.892704 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.892687    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:14:19.940738 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.940707    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:14:19.940738 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.940724    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:14:19.940738 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.940735    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:14:19.941030 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.940762    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Oct 30 19:14:19.941030 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.940794    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:14:19.941030 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.940888    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:14:19.941738 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.941717    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"]
Oct 30 19:14:19.954767 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.954737    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.954933 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.954817    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.954933 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.954859    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.954933 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.954893    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lks6w\" (UniqueName: \"kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.955125 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.954946    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.955186 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.955165    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.955674 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.955649    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.955769 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.955746    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:19.958178 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:19.958156    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:20.016056 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:20.016026    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lks6w\" (UniqueName: \"kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w\") pod \"isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") " pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:20.158052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:20.157974    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:20.574264 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:20.574046    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"]
Oct 30 19:14:20.577134 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:14:20.577105    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dcbe46_d851_4fa3_8357_9318debee3ed.slice/crio-d4a728d713a0a1c2ea537031823fef0b6d820e142c75880ccc812d90086e7775 WatchSource:0}: Error finding container d4a728d713a0a1c2ea537031823fef0b6d820e142c75880ccc812d90086e7775: Status 404 returned error can't find the container with id d4a728d713a0a1c2ea537031823fef0b6d820e142c75880ccc812d90086e7775
Oct 30 19:14:20.738152 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:20.738121    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerStarted","Data":"8e2234a2f59a1f3baaa0382c7c3672f6e8f6fbfe8ad30a27f3d88eab0213e397"}
Oct 30 19:14:20.738298 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:20.738176    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerStarted","Data":"d4a728d713a0a1c2ea537031823fef0b6d820e142c75880ccc812d90086e7775"}
Oct 30 19:14:22.870173 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:14:22.870138    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:14:25.759495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:25.759409    5972 generic.go:356] "Generic (PLEG): container finished" podID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerID="8e2234a2f59a1f3baaa0382c7c3672f6e8f6fbfe8ad30a27f3d88eab0213e397" exitCode=0
Oct 30 19:14:25.759495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:25.759482    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerDied","Data":"8e2234a2f59a1f3baaa0382c7c3672f6e8f6fbfe8ad30a27f3d88eab0213e397"}
Oct 30 19:14:26.765830 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:26.765800    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerStarted","Data":"6c9c932153841ba30fdc2562b1e58edc9801a2acf0d803f1c5254bb39c162e4d"}
Oct 30 19:14:26.766205 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:26.765854    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerStarted","Data":"f19e569c335db4b54128693fa3f0b251a57b94c302cdaa405f887ac1e7ea9408"}
Oct 30 19:14:26.766205 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:26.766131    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:26.766282 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:26.766266    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:26.767226 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:26.767206    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 19:14:26.884523 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:26.884453    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podStartSLOduration=3.029286714 podStartE2EDuration="7.884440281s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.855153567s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:26.883613355 +0000 UTC m=+2116.675580640" watchObservedRunningTime="2026-10-30 19:14:26.884440281 +0000 UTC m=+2116.676407569"
Oct 30 19:14:27.772908 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:27.772868    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 19:14:32.777475 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:32.777445    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:32.778067 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:32.778040    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 19:14:42.779023 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:14:42.778995    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:14:52.870470 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:14:52.870436    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:15:00.418694 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:00.418663    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"]
Oct 30 19:15:00.419114 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:00.418898    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container" containerID="cri-o://f19e569c335db4b54128693fa3f0b251a57b94c302cdaa405f887ac1e7ea9408" gracePeriod=30
Oct 30 19:15:00.419114 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:00.418927    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" containerID="cri-o://6c9c932153841ba30fdc2562b1e58edc9801a2acf0d803f1c5254bb39c162e4d" gracePeriod=30
Oct 30 19:15:00.906265 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:00.906231    5972 generic.go:356] "Generic (PLEG): container finished" podID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerID="6c9c932153841ba30fdc2562b1e58edc9801a2acf0d803f1c5254bb39c162e4d" exitCode=0
Oct 30 19:15:00.906430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:00.906303    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerDied","Data":"6c9c932153841ba30fdc2562b1e58edc9801a2acf0d803f1c5254bb39c162e4d"}
Oct 30 19:15:02.773763 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:02.773729    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 19:15:07.773198 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:07.773162    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 19:15:12.773651 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:12.773615    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 19:15:12.774072 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:12.773758    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:15:17.773958 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:17.773915    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 19:15:22.773710 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:22.773624    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 19:15:22.871796 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:15:22.871763    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:15:27.773074 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:27.773033    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 19:15:31.029727 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.029695    5972 generic.go:356] "Generic (PLEG): container finished" podID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerID="f19e569c335db4b54128693fa3f0b251a57b94c302cdaa405f887ac1e7ea9408" exitCode=137
Oct 30 19:15:31.030078 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.029767    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerDied","Data":"f19e569c335db4b54128693fa3f0b251a57b94c302cdaa405f887ac1e7ea9408"}
Oct 30 19:15:31.062374 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.062352    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:15:31.128845 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.128822    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location\") pod \"d6dcbe46-d851-4fa3-8357-9318debee3ed\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") "
Oct 30 19:15:31.128938 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.128862    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert\") pod \"d6dcbe46-d851-4fa3-8357-9318debee3ed\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") "
Oct 30 19:15:31.128938 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.128903    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls\") pod \"d6dcbe46-d851-4fa3-8357-9318debee3ed\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") "
Oct 30 19:15:31.129047 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.128940    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config\") pod \"d6dcbe46-d851-4fa3-8357-9318debee3ed\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") "
Oct 30 19:15:31.129047 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.129012    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w\" (UniqueName: \"kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w\") pod \"d6dcbe46-d851-4fa3-8357-9318debee3ed\" (UID: \"d6dcbe46-d851-4fa3-8357-9318debee3ed\") "
Oct 30 19:15:31.129348 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.129319    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert" pod "d6dcbe46-d851-4fa3-8357-9318debee3ed" (UID: "d6dcbe46-d851-4fa3-8357-9318debee3ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:31.129431 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.129331    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config" pod "d6dcbe46-d851-4fa3-8357-9318debee3ed" (UID: "d6dcbe46-d851-4fa3-8357-9318debee3ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:31.131174 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.131150    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls" pod "d6dcbe46-d851-4fa3-8357-9318debee3ed" (UID: "d6dcbe46-d851-4fa3-8357-9318debee3ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:31.131253 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.131221    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w" pod "d6dcbe46-d851-4fa3-8357-9318debee3ed" (UID: "d6dcbe46-d851-4fa3-8357-9318debee3ed"). InnerVolumeSpecName "kube-api-access-lks6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:31.139306 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.139285    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location" pod "d6dcbe46-d851-4fa3-8357-9318debee3ed" (UID: "d6dcbe46-d851-4fa3-8357-9318debee3ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:31.230518 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.230446    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lks6w\" (UniqueName: \"kubernetes.io/projected/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-api-access-lks6w\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:31.230518 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.230489    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6dcbe46-d851-4fa3-8357-9318debee3ed-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:31.230518 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.230509    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:31.230738 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.230526    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6dcbe46-d851-4fa3-8357-9318debee3ed-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:31.230738 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:31.230546    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6dcbe46-d851-4fa3-8357-9318debee3ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:32.035248 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.035215    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z" event={"ID":"d6dcbe46-d851-4fa3-8357-9318debee3ed","Type":"ContainerDied","Data":"d4a728d713a0a1c2ea537031823fef0b6d820e142c75880ccc812d90086e7775"}
Oct 30 19:15:32.035248 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.035231    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"
Oct 30 19:15:32.035821 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.035280    5972 scope.go:122] "RemoveContainer" containerID="6c9c932153841ba30fdc2562b1e58edc9801a2acf0d803f1c5254bb39c162e4d"
Oct 30 19:15:32.043793 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.043778    5972 scope.go:122] "RemoveContainer" containerID="f19e569c335db4b54128693fa3f0b251a57b94c302cdaa405f887ac1e7ea9408"
Oct 30 19:15:32.051028 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.051011    5972 scope.go:122] "RemoveContainer" containerID="8e2234a2f59a1f3baaa0382c7c3672f6e8f6fbfe8ad30a27f3d88eab0213e397"
Oct 30 19:15:32.242111 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.242082    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"]
Oct 30 19:15:32.290203 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.290141    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-tensorflow-runtime-predictor-86bbf9b864-xrq2z"]
Oct 30 19:15:32.338303 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:32.338274    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" path="/var/lib/kubelet/pods/d6dcbe46-d851-4fa3-8357-9318debee3ed/volumes"
Oct 30 19:15:35.897049 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897012    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"]
Oct 30 19:15:35.897521 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897478    5972 cpu_manager.go:414] "containerMap: removing container" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy"
Oct 30 19:15:35.897521 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897495    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy"
Oct 30 19:15:35.897677 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897534    5972 cpu_manager.go:414] "containerMap: removing container" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container"
Oct 30 19:15:35.897677 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897542    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container"
Oct 30 19:15:35.897677 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897559    5972 cpu_manager.go:414] "containerMap: removing container" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="storage-initializer"
Oct 30 19:15:35.897677 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897568    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="storage-initializer"
Oct 30 19:15:35.897882 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897685    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kube-rbac-proxy"
Oct 30 19:15:35.897882 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.897702    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6dcbe46-d851-4fa3-8357-9318debee3ed" containerName="kserve-container"
Oct 30 19:15:35.902315 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.902293    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:35.953798 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.953773    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Oct 30 19:15:35.953949 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.953931    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:15:35.954049 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.954035    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:15:35.961385 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.961363    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:35.961462 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.961394    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvtv\" (UniqueName: \"kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:35.961462 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.961415    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:35.961462 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.961431    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:35.961626 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:35.961605    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.001690 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.001657    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-triton-predictor-serving-cert\""
Oct 30 19:15:36.001875 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.001800    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:15:36.001875 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.001813    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:15:36.001875 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.001839    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:15:36.002806 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.002790    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"]
Oct 30 19:15:36.062492 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.062472    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.062624 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.062534    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.062624 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.062561    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvtv\" (UniqueName: \"kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.062624 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.062614    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.062770 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.062638    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.063109 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.063089    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.063399 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.063375    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.063441 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.063375    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.065069 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.065053    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.122074 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.122049    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvtv\" (UniqueName: \"kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv\") pod \"isvc-triton-predictor-7bcf87669-qbgxn\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") " pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.218250 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.218212    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:15:36.635182 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:36.635113    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"]
Oct 30 19:15:36.638631 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:15:36.638602    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e0730c_9b81_4042_89a2_59e569ddc08e.slice/crio-9c7696e22caf4a595b007910454a1f9b2628e2a7de9b507f23669877eeb42e20 WatchSource:0}: Error finding container 9c7696e22caf4a595b007910454a1f9b2628e2a7de9b507f23669877eeb42e20: Status 404 returned error can't find the container with id 9c7696e22caf4a595b007910454a1f9b2628e2a7de9b507f23669877eeb42e20
Oct 30 19:15:37.058006 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:37.057971    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerStarted","Data":"8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003"}
Oct 30 19:15:37.058503 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:37.058027    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerStarted","Data":"9c7696e22caf4a595b007910454a1f9b2628e2a7de9b507f23669877eeb42e20"}
Oct 30 19:15:41.075829 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:41.075795    5972 generic.go:356] "Generic (PLEG): container finished" podID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerID="8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003" exitCode=0
Oct 30 19:15:41.076270 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:15:41.075868    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerDied","Data":"8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003"}
Oct 30 19:15:52.913257 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:15:52.913178    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:16:22.878262 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:16:22.878221    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:16:52.873916 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:16:52.873833    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:17:22.875164 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:17:22.875125    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:17:36.641104 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:36.641068    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerStarted","Data":"c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453"}
Oct 30 19:17:36.641723 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:36.641125    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerStarted","Data":"662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386"}
Oct 30 19:17:36.641723 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:36.641233    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:17:36.757245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:36.757194    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" podStartSLOduration=3.337527135 podStartE2EDuration="2m1.757181024s" totalImagesPullingTime="1m54.502252523s" totalInitContainerRuntime="3.917401366s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:36.756551552 +0000 UTC m=+2306.548518833" watchObservedRunningTime="2026-10-30 19:17:36.757181024 +0000 UTC m=+2306.549148312"
Oct 30 19:17:37.644733 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:37.644701    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:17:37.645805 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:37.645778    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 19:17:38.648680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:38.648636    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 19:17:43.653674 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:43.653646    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:17:43.654497 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:43.654479    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:17:52.869629 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:17:52.869575    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:17:58.621435 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:58.621399    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"]
Oct 30 19:17:58.621897 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:58.621669    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kserve-container" containerID="cri-o://662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386" gracePeriod=30
Oct 30 19:17:58.621897 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:58.621701    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kube-rbac-proxy" containerID="cri-o://c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453" gracePeriod=30
Oct 30 19:17:58.649570 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:58.649543    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Oct 30 19:17:59.730084 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:59.730055    5972 generic.go:356] "Generic (PLEG): container finished" podID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerID="c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453" exitCode=0
Oct 30 19:17:59.730447 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:17:59.730109    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerDied","Data":"c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453"}
Oct 30 19:18:01.396876 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.396841    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:18:01.495309 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495246    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert\") pod \"a1e0730c-9b81-4042-89a2-59e569ddc08e\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") "
Oct 30 19:18:01.495309 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495295    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls\") pod \"a1e0730c-9b81-4042-89a2-59e569ddc08e\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") "
Oct 30 19:18:01.495508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495331    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv\" (UniqueName: \"kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv\") pod \"a1e0730c-9b81-4042-89a2-59e569ddc08e\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") "
Oct 30 19:18:01.495508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495356    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config\") pod \"a1e0730c-9b81-4042-89a2-59e569ddc08e\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") "
Oct 30 19:18:01.495508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495377    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location\") pod \"a1e0730c-9b81-4042-89a2-59e569ddc08e\" (UID: \"a1e0730c-9b81-4042-89a2-59e569ddc08e\") "
Oct 30 19:18:01.495791 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495647    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert" pod "a1e0730c-9b81-4042-89a2-59e569ddc08e" (UID: "a1e0730c-9b81-4042-89a2-59e569ddc08e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:01.495906 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495810    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config" pod "a1e0730c-9b81-4042-89a2-59e569ddc08e" (UID: "a1e0730c-9b81-4042-89a2-59e569ddc08e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:01.495972 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.495942    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location" pod "a1e0730c-9b81-4042-89a2-59e569ddc08e" (UID: "a1e0730c-9b81-4042-89a2-59e569ddc08e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:01.497602 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.497567    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls" pod "a1e0730c-9b81-4042-89a2-59e569ddc08e" (UID: "a1e0730c-9b81-4042-89a2-59e569ddc08e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:01.497718 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.497700    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv" pod "a1e0730c-9b81-4042-89a2-59e569ddc08e" (UID: "a1e0730c-9b81-4042-89a2-59e569ddc08e"). InnerVolumeSpecName "kube-api-access-7tvtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:01.596075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.596045    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e0730c-9b81-4042-89a2-59e569ddc08e-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.596075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.596068    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.596075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.596079    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e0730c-9b81-4042-89a2-59e569ddc08e-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.596281 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.596089    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvtv\" (UniqueName: \"kubernetes.io/projected/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-api-access-7tvtv\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.596281 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.596098    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e0730c-9b81-4042-89a2-59e569ddc08e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.740188 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.740154    5972 generic.go:356] "Generic (PLEG): container finished" podID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerID="662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386" exitCode=0
Oct 30 19:18:01.740333 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.740215    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerDied","Data":"662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386"}
Oct 30 19:18:01.740333 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.740273    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"
Oct 30 19:18:01.740333 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.740299    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn" event={"ID":"a1e0730c-9b81-4042-89a2-59e569ddc08e","Type":"ContainerDied","Data":"9c7696e22caf4a595b007910454a1f9b2628e2a7de9b507f23669877eeb42e20"}
Oct 30 19:18:01.740333 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.740325    5972 scope.go:122] "RemoveContainer" containerID="c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453"
Oct 30 19:18:01.749451 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.749434    5972 scope.go:122] "RemoveContainer" containerID="662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386"
Oct 30 19:18:01.756737 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.756719    5972 scope.go:122] "RemoveContainer" containerID="8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003"
Oct 30 19:18:01.763761 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.763747    5972 scope.go:122] "RemoveContainer" containerID="c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453"
Oct 30 19:18:01.764044 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:18:01.764026    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453\": container with ID starting with c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453 not found: ID does not exist" containerID="c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453"
Oct 30 19:18:01.764091 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.764055    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453"} err="failed to get container status \"c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453\": rpc error: code = NotFound desc = could not find container \"c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453\": container with ID starting with c579201c2fd17e21d6ddb55a2dd7f113daf654b72db779d26298404716d5f453 not found: ID does not exist"
Oct 30 19:18:01.764091 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.764073    5972 scope.go:122] "RemoveContainer" containerID="662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386"
Oct 30 19:18:01.764302 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:18:01.764284    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386\": container with ID starting with 662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386 not found: ID does not exist" containerID="662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386"
Oct 30 19:18:01.764366 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.764310    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386"} err="failed to get container status \"662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386\": rpc error: code = NotFound desc = could not find container \"662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386\": container with ID starting with 662c83c0bb08c4a96c17a3ed3f6800a7d6c861ff3d531edb264efafcb56fe386 not found: ID does not exist"
Oct 30 19:18:01.764366 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.764331    5972 scope.go:122] "RemoveContainer" containerID="8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003"
Oct 30 19:18:01.764555 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:18:01.764539    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003\": container with ID starting with 8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003 not found: ID does not exist" containerID="8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003"
Oct 30 19:18:01.764607 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.764558    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003"} err="failed to get container status \"8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003\": rpc error: code = NotFound desc = could not find container \"8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003\": container with ID starting with 8fa9317fe7a42a3cb32a3075d0a036c9cf3c7ae839e79bfdf256baf367385003 not found: ID does not exist"
Oct 30 19:18:01.899466 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.899444    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"]
Oct 30 19:18:01.903234 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:01.903216    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-triton-predictor-7bcf87669-qbgxn"]
Oct 30 19:18:02.338656 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:02.338628    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" path="/var/lib/kubelet/pods/a1e0730c-9b81-4042-89a2-59e569ddc08e/volumes"
Oct 30 19:18:05.640739 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.640708    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"]
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641054    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="storage-initializer"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641066    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="storage-initializer"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641081    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kserve-container"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641086    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kserve-container"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641101    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kube-rbac-proxy"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641106    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kube-rbac-proxy"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641166    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kserve-container"
Oct 30 19:18:05.641222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.641177    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e0730c-9b81-4042-89a2-59e569ddc08e" containerName="kube-rbac-proxy"
Oct 30 19:18:05.658211 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.658188    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.709589 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.709560    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 19:18:05.709763 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.709671    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:18:05.729694 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.729666    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd457\" (UniqueName: \"kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.729784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.729698    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.729784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.729714    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.729784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.729777    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.729881 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.729862    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.756833 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.756812    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:05.756970 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.756936    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:05.757051 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.756963    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:18:05.757051 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.756976    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:18:05.757051 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.757030    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-predictor-serving-cert\""
Oct 30 19:18:05.758223 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.758208    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"]
Oct 30 19:18:05.830785 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.830756    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd457\" (UniqueName: \"kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.830785 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.830785    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.830945 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.830801    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.830997 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.830978    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.831058 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.831033    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.831175 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.831161    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.831469 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.831451    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.831627 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.831612    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.833535 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.833519    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:05.890563 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:05.890533    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd457\" (UniqueName: \"kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457\") pod \"isvc-xgboost-predictor-6f67855d76-h267p\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:06.031022 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:06.031001    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:06.463122 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:06.463100    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"]
Oct 30 19:18:06.467430 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:18:06.467385    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9417ec7_5567_4f09_bf4c_a38dc1eba8b3.slice/crio-e8434275a62221aa47082074d4c7cdf8a500a6135e27720d26e95d9b3ec26b24 WatchSource:0}: Error finding container e8434275a62221aa47082074d4c7cdf8a500a6135e27720d26e95d9b3ec26b24: Status 404 returned error can't find the container with id e8434275a62221aa47082074d4c7cdf8a500a6135e27720d26e95d9b3ec26b24
Oct 30 19:18:06.764009 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:06.763934    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerStarted","Data":"47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d"}
Oct 30 19:18:06.764009 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:06.763994    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerStarted","Data":"e8434275a62221aa47082074d4c7cdf8a500a6135e27720d26e95d9b3ec26b24"}
Oct 30 19:18:10.780416 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:10.780380    5972 generic.go:356] "Generic (PLEG): container finished" podID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerID="47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d" exitCode=0
Oct 30 19:18:10.780791 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:10.780443    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerDied","Data":"47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d"}
Oct 30 19:18:10.781662 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:10.781646    5972 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:18:22.873328 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:18:22.873242    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:18:30.871695 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:30.871657    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerStarted","Data":"80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab"}
Oct 30 19:18:30.872186 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:30.871729    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerStarted","Data":"979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095"}
Oct 30 19:18:30.872186 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:30.871960    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:31.037208 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:31.037164    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podStartSLOduration=2.878147223 podStartE2EDuration="26.037152005s" totalImagesPullingTime="19.493120688s" totalInitContainerRuntime="3.665884094s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:31.036709427 +0000 UTC m=+2360.828676712" watchObservedRunningTime="2026-10-30 19:18:31.037152005 +0000 UTC m=+2360.829119293"
Oct 30 19:18:31.876060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:31.876034    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:31.877322 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:31.877297    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:18:32.879812 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:32.879774    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:18:37.884646 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:37.884616    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:18:37.885223 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:37.885176    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:18:47.885193 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:47.885153    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:18:52.869782 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:18:52.869741    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:18:57.885343 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:18:57.885292    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:19:07.885108 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:07.885072    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:19:10.595837 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:10.595802    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:19:10.613503 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:10.613481    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:19:16.240794 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.240760    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"]
Oct 30 19:19:16.243688 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.243656    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.294546 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.294509    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 19:19:16.294745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.294552    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:19:16.294745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.294559    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:19:16.345100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.345063    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:19:16.345100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.345100    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:19:16.345100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.345063    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:19:16.345387 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.345148    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Oct 30 19:19:16.345811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.345784    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.345951 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.345834    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.346112 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.346052    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.346287 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.346266    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpssz\" (UniqueName: \"kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.346376 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.346315    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.347388 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.347368    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"]
Oct 30 19:19:16.447485 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.447454    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpssz\" (UniqueName: \"kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.447485 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.447498    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.447759 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.447541    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.447759 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.447563    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.447759 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.447651    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.448403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.448367    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.448514 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.448369    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.448514 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.448474    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.450277 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.450259    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.507057 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.506999    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpssz\" (UniqueName: \"kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz\") pod \"isvc-canary-create-v2-predictor-546fbc85f4-b8f5l\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.559098 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.559074    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:16.783067 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.782996    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"]
Oct 30 19:19:16.786107 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.786088    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.836761 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.836728    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 19:19:16.843427 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.843403    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"]
Oct 30 19:19:16.850279 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.850258    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.850400 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.850295    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.850400 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.850321    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.850525 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.850453    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.850525 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.850515    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbp7\" (UniqueName: \"kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.951641 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.951609    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbp7\" (UniqueName: \"kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.951804 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.951664    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.951804 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.951696    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.951804 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.951724    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.951804 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.951750    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.952140 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.952116    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.952545 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.952524    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.952598 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.952524    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.954341 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.954319    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:16.962772 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:16.962749    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"]
Oct 30 19:19:16.965630 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:19:16.965609    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3b5362_7d2f_47de_8384_1755f88403f1.slice/crio-82dd48c1b46a63daa0153e95664f14e9137d588190c6f5de56bc4c9ae65639eb WatchSource:0}: Error finding container 82dd48c1b46a63daa0153e95664f14e9137d588190c6f5de56bc4c9ae65639eb: Status 404 returned error can't find the container with id 82dd48c1b46a63daa0153e95664f14e9137d588190c6f5de56bc4c9ae65639eb
Oct 30 19:19:17.011193 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:17.011173    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbp7\" (UniqueName: \"kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7\") pod \"isvc-canary-create-predictor-657958dc5c-gd7dv\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:17.050868 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:17.050803    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerStarted","Data":"101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4"}
Oct 30 19:19:17.050868 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:17.050860    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerStarted","Data":"82dd48c1b46a63daa0153e95664f14e9137d588190c6f5de56bc4c9ae65639eb"}
Oct 30 19:19:17.099996 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:17.099974    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:17.472323 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:17.472293    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"]
Oct 30 19:19:17.475141 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:19:17.475112    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78af202f_ea59_4685_8cac_8926806022ce.slice/crio-5017f9637bae81ca92b0b32d6a0a771a5231a9e16d202a0b847f83e36f2c422e WatchSource:0}: Error finding container 5017f9637bae81ca92b0b32d6a0a771a5231a9e16d202a0b847f83e36f2c422e: Status 404 returned error can't find the container with id 5017f9637bae81ca92b0b32d6a0a771a5231a9e16d202a0b847f83e36f2c422e
Oct 30 19:19:17.885974 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:17.885887    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:19:18.056654 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:18.056618    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerStarted","Data":"97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d"}
Oct 30 19:19:18.056828 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:18.056694    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerStarted","Data":"5017f9637bae81ca92b0b32d6a0a771a5231a9e16d202a0b847f83e36f2c422e"}
Oct 30 19:19:21.070234 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:21.070201    5972 generic.go:356] "Generic (PLEG): container finished" podID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerID="101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4" exitCode=0
Oct 30 19:19:21.070569 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:21.070277    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerDied","Data":"101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4"}
Oct 30 19:19:22.077232 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:22.077192    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerStarted","Data":"23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4"}
Oct 30 19:19:22.077709 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:22.077270    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerStarted","Data":"7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf"}
Oct 30 19:19:22.077709 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:22.077558    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:22.078690 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:22.078670    5972 generic.go:356] "Generic (PLEG): container finished" podID="78af202f-ea59-4685-8cac-8926806022ce" containerID="97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d" exitCode=0
Oct 30 19:19:22.078789 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:22.078719    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerDied","Data":"97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d"}
Oct 30 19:19:22.218364 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:22.218321    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podStartSLOduration=2.252560028 podStartE2EDuration="6.218308804s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.965748776s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.217264721 +0000 UTC m=+2412.009232023" watchObservedRunningTime="2026-10-30 19:19:22.218308804 +0000 UTC m=+2412.010276092"
Oct 30 19:19:22.868908 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:19:22.868873    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:19:23.084984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:23.084953    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerStarted","Data":"4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa"}
Oct 30 19:19:23.085430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:23.085009    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerStarted","Data":"00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd"}
Oct 30 19:19:23.085430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:23.085203    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:23.085430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:23.085412    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:23.086518 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:23.086488    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:19:23.208069 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:23.208025    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podStartSLOduration=3.27123773 podStartE2EDuration="7.208011718s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.936773988s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:23.207976057 +0000 UTC m=+2412.999943345" watchObservedRunningTime="2026-10-30 19:19:23.208011718 +0000 UTC m=+2412.999979005"
Oct 30 19:19:24.088649 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:24.088611    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:19:24.089034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:24.088847    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:24.090060 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:24.090036    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:19:25.092497 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:25.092456    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:19:27.885242 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:27.885202    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:19:29.093128 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:29.093101    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:19:29.093609 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:29.093561    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:19:30.097161 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:30.097121    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:19:30.097680 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:30.097650    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:19:37.885646 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:37.885612    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:19:39.094036 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:39.093997    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:19:40.097818 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:40.097783    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:19:46.338821 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:46.338781    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"]
Oct 30 19:19:46.339326 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:46.339091    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" containerID="cri-o://979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095" gracePeriod=30
Oct 30 19:19:46.339326 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:46.339146    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kube-rbac-proxy" containerID="cri-o://80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab" gracePeriod=30
Oct 30 19:19:47.183496 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:47.183454    5972 generic.go:356] "Generic (PLEG): container finished" podID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerID="80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab" exitCode=0
Oct 30 19:19:47.183685 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:47.183527    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerDied","Data":"80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab"}
Oct 30 19:19:47.880244 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:47.880204    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Oct 30 19:19:47.885557 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:47.885533    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 19:19:49.093546 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:49.093468    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:19:50.098248 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.098211    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:19:50.382410 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.382384    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:19:50.542286 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542255    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457\" (UniqueName: \"kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457\") pod \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") "
Oct 30 19:19:50.542468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542305    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls\") pod \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") "
Oct 30 19:19:50.542468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542344    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert\") pod \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") "
Oct 30 19:19:50.542468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542375    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location\") pod \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") "
Oct 30 19:19:50.542468 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542423    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config\") pod \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\" (UID: \"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3\") "
Oct 30 19:19:50.542801 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542770    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert" pod "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" (UID: "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:50.542801 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542769    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location" pod "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" (UID: "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:50.542956 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.542873    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config" pod "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" (UID: "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:50.544688 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.544668    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls" pod "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" (UID: "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:50.544757 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.544664    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457" pod "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" (UID: "b9417ec7-5567-4f09-bf4c-a38dc1eba8b3"). InnerVolumeSpecName "kube-api-access-kd457". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:50.643847 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.643780    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd457\" (UniqueName: \"kubernetes.io/projected/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-api-access-kd457\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.643847 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.643802    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.643847 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.643812    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.643847 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.643821    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.643847 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:50.643829    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:51.202574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.202542    5972 generic.go:356] "Generic (PLEG): container finished" podID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerID="979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095" exitCode=0
Oct 30 19:19:51.202574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.202604    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerDied","Data":"979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095"}
Oct 30 19:19:51.203151 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.202616    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"
Oct 30 19:19:51.203151 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.202661    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p" event={"ID":"b9417ec7-5567-4f09-bf4c-a38dc1eba8b3","Type":"ContainerDied","Data":"e8434275a62221aa47082074d4c7cdf8a500a6135e27720d26e95d9b3ec26b24"}
Oct 30 19:19:51.203151 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.202691    5972 scope.go:122] "RemoveContainer" containerID="80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab"
Oct 30 19:19:51.213771 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.213745    5972 scope.go:122] "RemoveContainer" containerID="979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095"
Oct 30 19:19:51.222543 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.222525    5972 scope.go:122] "RemoveContainer" containerID="47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d"
Oct 30 19:19:51.230071 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.230054    5972 scope.go:122] "RemoveContainer" containerID="80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab"
Oct 30 19:19:51.230335 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:19:51.230314    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab\": container with ID starting with 80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab not found: ID does not exist" containerID="80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab"
Oct 30 19:19:51.230413 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.230347    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab"} err="failed to get container status \"80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab\": rpc error: code = NotFound desc = could not find container \"80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab\": container with ID starting with 80d62d1dbc0b483b6e7554fba9d6ae9ccc8cfa26a83d00d939dd973cc67619ab not found: ID does not exist"
Oct 30 19:19:51.230413 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.230372    5972 scope.go:122] "RemoveContainer" containerID="979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095"
Oct 30 19:19:51.230615 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:19:51.230598    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095\": container with ID starting with 979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095 not found: ID does not exist" containerID="979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095"
Oct 30 19:19:51.230688 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.230624    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095"} err="failed to get container status \"979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095\": rpc error: code = NotFound desc = could not find container \"979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095\": container with ID starting with 979ffe03a26aa8b359af550176f66f710752389c387effe1d8af4a107f474095 not found: ID does not exist"
Oct 30 19:19:51.230688 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.230643    5972 scope.go:122] "RemoveContainer" containerID="47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d"
Oct 30 19:19:51.230864 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:19:51.230847    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d\": container with ID starting with 47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d not found: ID does not exist" containerID="47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d"
Oct 30 19:19:51.230904 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.230869    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d"} err="failed to get container status \"47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d\": rpc error: code = NotFound desc = could not find container \"47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d\": container with ID starting with 47aca6958a35909a71baad7657e084a838247aa5fd043f16d620961de10be65d not found: ID does not exist"
Oct 30 19:19:51.410193 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.410162    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"]
Oct 30 19:19:51.414953 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:51.414933    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-predictor-6f67855d76-h267p"]
Oct 30 19:19:52.338941 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:52.338911    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" path="/var/lib/kubelet/pods/b9417ec7-5567-4f09-bf4c-a38dc1eba8b3/volumes"
Oct 30 19:19:52.870459 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:19:52.870426    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:19:59.094016 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:19:59.093976    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:20:00.098243 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:00.098200    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:20:09.094416 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:09.094377    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:20:10.098218 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:10.098174    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:20:19.094326 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:19.094292    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:20:20.097729 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:20.097672    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:20:22.869927 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:22.869890    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:20:29.093765 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:29.093725    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:20:30.098408 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:30.098374    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:20:39.094728 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:39.094700    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:20:40.098733 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:40.098702    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:20:46.096360 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.096327    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"]
Oct 30 19:20:46.096850 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.096568    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" containerID="cri-o://00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd" gracePeriod=30
Oct 30 19:20:46.096850 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.096665    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kube-rbac-proxy" containerID="cri-o://4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa" gracePeriod=30
Oct 30 19:20:46.097328 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.097171    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"]
Oct 30 19:20:46.097417 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.097382    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" containerID="cri-o://7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf" gracePeriod=30
Oct 30 19:20:46.097480 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.097408    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kube-rbac-proxy" containerID="cri-o://23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4" gracePeriod=30
Oct 30 19:20:46.423489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.423394    5972 generic.go:356] "Generic (PLEG): container finished" podID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerID="23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4" exitCode=0
Oct 30 19:20:46.423489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.423468    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerDied","Data":"23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4"}
Oct 30 19:20:46.425694 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.425673    5972 generic.go:356] "Generic (PLEG): container finished" podID="78af202f-ea59-4685-8cac-8926806022ce" containerID="4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa" exitCode=0
Oct 30 19:20:46.425813 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:46.425708    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerDied","Data":"4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa"}
Oct 30 19:20:49.089260 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:49.089226    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Oct 30 19:20:49.093576 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:49.093554    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 19:20:50.092791 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:50.092747    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Oct 30 19:20:50.098154 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:50.098131    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 19:20:50.932953 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:50.932928    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:20:51.034056 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.034030    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:20:51.055181 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055164    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert\") pod \"78af202f-ea59-4685-8cac-8926806022ce\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") "
Oct 30 19:20:51.055271 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055210    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config\") pod \"78af202f-ea59-4685-8cac-8926806022ce\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") "
Oct 30 19:20:51.055271 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055237    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7\" (UniqueName: \"kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7\") pod \"78af202f-ea59-4685-8cac-8926806022ce\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") "
Oct 30 19:20:51.055378 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055307    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location\") pod \"78af202f-ea59-4685-8cac-8926806022ce\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") "
Oct 30 19:20:51.055472 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055450    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls\") pod \"78af202f-ea59-4685-8cac-8926806022ce\" (UID: \"78af202f-ea59-4685-8cac-8926806022ce\") "
Oct 30 19:20:51.055612 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055565    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert" pod "78af202f-ea59-4685-8cac-8926806022ce" (UID: "78af202f-ea59-4685-8cac-8926806022ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:51.055696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055620    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location" pod "78af202f-ea59-4685-8cac-8926806022ce" (UID: "78af202f-ea59-4685-8cac-8926806022ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:51.055696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055660    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config" pod "78af202f-ea59-4685-8cac-8926806022ce" (UID: "78af202f-ea59-4685-8cac-8926806022ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:51.055813 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055795    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78af202f-ea59-4685-8cac-8926806022ce-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.055866 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055820    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.055866 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.055836    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af202f-ea59-4685-8cac-8926806022ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.057404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.057385    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7" pod "78af202f-ea59-4685-8cac-8926806022ce" (UID: "78af202f-ea59-4685-8cac-8926806022ce"). InnerVolumeSpecName "kube-api-access-4gbp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:51.057452 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.057424    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls" pod "78af202f-ea59-4685-8cac-8926806022ce" (UID: "78af202f-ea59-4685-8cac-8926806022ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:51.156058 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156028    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz\" (UniqueName: \"kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz\") pod \"4d3b5362-7d2f-47de-8384-1755f88403f1\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") "
Oct 30 19:20:51.156456 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156092    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls\") pod \"4d3b5362-7d2f-47de-8384-1755f88403f1\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") "
Oct 30 19:20:51.156456 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156120    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config\") pod \"4d3b5362-7d2f-47de-8384-1755f88403f1\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") "
Oct 30 19:20:51.156456 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156145    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert\") pod \"4d3b5362-7d2f-47de-8384-1755f88403f1\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") "
Oct 30 19:20:51.156456 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156182    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location\") pod \"4d3b5362-7d2f-47de-8384-1755f88403f1\" (UID: \"4d3b5362-7d2f-47de-8384-1755f88403f1\") "
Oct 30 19:20:51.156456 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156415    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbp7\" (UniqueName: \"kubernetes.io/projected/78af202f-ea59-4685-8cac-8926806022ce-kube-api-access-4gbp7\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.156456 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156436    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af202f-ea59-4685-8cac-8926806022ce-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.156745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156533    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert" pod "4d3b5362-7d2f-47de-8384-1755f88403f1" (UID: "4d3b5362-7d2f-47de-8384-1755f88403f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:51.156745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156535    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config" pod "4d3b5362-7d2f-47de-8384-1755f88403f1" (UID: "4d3b5362-7d2f-47de-8384-1755f88403f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:51.156745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.156576    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location" pod "4d3b5362-7d2f-47de-8384-1755f88403f1" (UID: "4d3b5362-7d2f-47de-8384-1755f88403f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:51.158100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.158085    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz" pod "4d3b5362-7d2f-47de-8384-1755f88403f1" (UID: "4d3b5362-7d2f-47de-8384-1755f88403f1"). InnerVolumeSpecName "kube-api-access-vpssz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:51.158238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.158221    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls" pod "4d3b5362-7d2f-47de-8384-1755f88403f1" (UID: "4d3b5362-7d2f-47de-8384-1755f88403f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:51.257811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.257745    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.257811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.257769    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d3b5362-7d2f-47de-8384-1755f88403f1-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.257811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.257780    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpssz\" (UniqueName: \"kubernetes.io/projected/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-api-access-vpssz\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.257811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.257795    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d3b5362-7d2f-47de-8384-1755f88403f1-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.257811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.257807    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d3b5362-7d2f-47de-8384-1755f88403f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.449077 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.449042    5972 generic.go:356] "Generic (PLEG): container finished" podID="78af202f-ea59-4685-8cac-8926806022ce" containerID="00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd" exitCode=0
Oct 30 19:20:51.449245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.449100    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerDied","Data":"00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd"}
Oct 30 19:20:51.449245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.449115    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"
Oct 30 19:20:51.449245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.449161    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv" event={"ID":"78af202f-ea59-4685-8cac-8926806022ce","Type":"ContainerDied","Data":"5017f9637bae81ca92b0b32d6a0a771a5231a9e16d202a0b847f83e36f2c422e"}
Oct 30 19:20:51.449245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.449199    5972 scope.go:122] "RemoveContainer" containerID="4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa"
Oct 30 19:20:51.451321 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.451285    5972 generic.go:356] "Generic (PLEG): container finished" podID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerID="7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf" exitCode=0
Oct 30 19:20:51.451429 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.451337    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerDied","Data":"7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf"}
Oct 30 19:20:51.451429 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.451404    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l" event={"ID":"4d3b5362-7d2f-47de-8384-1755f88403f1","Type":"ContainerDied","Data":"82dd48c1b46a63daa0153e95664f14e9137d588190c6f5de56bc4c9ae65639eb"}
Oct 30 19:20:51.451528 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.451341    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"
Oct 30 19:20:51.459139 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.459123    5972 scope.go:122] "RemoveContainer" containerID="00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd"
Oct 30 19:20:51.466608 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.466592    5972 scope.go:122] "RemoveContainer" containerID="97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d"
Oct 30 19:20:51.475487 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.475457    5972 scope.go:122] "RemoveContainer" containerID="4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa"
Oct 30 19:20:51.475786 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:51.475758    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa\": container with ID starting with 4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa not found: ID does not exist" containerID="4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa"
Oct 30 19:20:51.475871 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.475798    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa"} err="failed to get container status \"4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa\": rpc error: code = NotFound desc = could not find container \"4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa\": container with ID starting with 4e74d1c25c9e75972192ce472685c66761d5998b65b345cec92fb8244665f6aa not found: ID does not exist"
Oct 30 19:20:51.475871 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.475818    5972 scope.go:122] "RemoveContainer" containerID="00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd"
Oct 30 19:20:51.476065 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:51.476048    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd\": container with ID starting with 00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd not found: ID does not exist" containerID="00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd"
Oct 30 19:20:51.476112 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.476071    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd"} err="failed to get container status \"00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd\": rpc error: code = NotFound desc = could not find container \"00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd\": container with ID starting with 00b14902ff08161d001243dc2dc438e121d024e4a4b07ff0c1e5b255582859bd not found: ID does not exist"
Oct 30 19:20:51.476112 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.476086    5972 scope.go:122] "RemoveContainer" containerID="97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d"
Oct 30 19:20:51.476291 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:51.476276    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d\": container with ID starting with 97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d not found: ID does not exist" containerID="97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d"
Oct 30 19:20:51.476329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.476297    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d"} err="failed to get container status \"97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d\": rpc error: code = NotFound desc = could not find container \"97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d\": container with ID starting with 97cfb825910c2fa590b4eb30b80994681b5133297e1c68799e2c60da3198f42d not found: ID does not exist"
Oct 30 19:20:51.476329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.476310    5972 scope.go:122] "RemoveContainer" containerID="23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4"
Oct 30 19:20:51.483391 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.483376    5972 scope.go:122] "RemoveContainer" containerID="7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf"
Oct 30 19:20:51.490358 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.490340    5972 scope.go:122] "RemoveContainer" containerID="101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4"
Oct 30 19:20:51.497389 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.497336    5972 scope.go:122] "RemoveContainer" containerID="23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4"
Oct 30 19:20:51.497623 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:51.497607    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4\": container with ID starting with 23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4 not found: ID does not exist" containerID="23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4"
Oct 30 19:20:51.497686 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.497627    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4"} err="failed to get container status \"23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4\": rpc error: code = NotFound desc = could not find container \"23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4\": container with ID starting with 23c493bdeeaabebc49777c4fd84ee0ccb7aa76fc3822f4df6ccce17e369600e4 not found: ID does not exist"
Oct 30 19:20:51.497686 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.497640    5972 scope.go:122] "RemoveContainer" containerID="7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf"
Oct 30 19:20:51.497847 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:51.497831    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf\": container with ID starting with 7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf not found: ID does not exist" containerID="7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf"
Oct 30 19:20:51.497882 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.497851    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf"} err="failed to get container status \"7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf\": rpc error: code = NotFound desc = could not find container \"7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf\": container with ID starting with 7b9e47b0fda566c1391393346d15392a4540f8598de3b49be7ee4630278eb2cf not found: ID does not exist"
Oct 30 19:20:51.497882 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.497864    5972 scope.go:122] "RemoveContainer" containerID="101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4"
Oct 30 19:20:51.498051 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:51.498037    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4\": container with ID starting with 101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4 not found: ID does not exist" containerID="101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4"
Oct 30 19:20:51.498093 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.498054    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4"} err="failed to get container status \"101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4\": rpc error: code = NotFound desc = could not find container \"101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4\": container with ID starting with 101917cc6bfa692745465f05f99eaeae109e6048b422d47dccc3a921c793aef4 not found: ID does not exist"
Oct 30 19:20:51.608647 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.608568    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"]
Oct 30 19:20:51.656201 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.656173    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-gd7dv"]
Oct 30 19:20:51.721234 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.721208    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"]
Oct 30 19:20:51.725345 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:51.725325    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-v2-predictor-546fbc85f4-b8f5l"]
Oct 30 19:20:52.338212 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:52.338180    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" path="/var/lib/kubelet/pods/4d3b5362-7d2f-47de-8384-1755f88403f1/volumes"
Oct 30 19:20:52.338692 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:20:52.338678    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78af202f-ea59-4685-8cac-8926806022ce" path="/var/lib/kubelet/pods/78af202f-ea59-4685-8cac-8926806022ce/volumes"
Oct 30 19:20:52.870038 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:20:52.870003    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:21:15.317217 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317185    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"]
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317519    5972 cpu_manager.go:414] "containerMap: removing container" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="storage-initializer"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317531    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="storage-initializer"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317550    5972 cpu_manager.go:414] "containerMap: removing container" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317559    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317576    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317595    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317604    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="storage-initializer"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317609    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="storage-initializer"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317615    5972 cpu_manager.go:414] "containerMap: removing container" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="storage-initializer"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317619    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="storage-initializer"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317625    5972 cpu_manager.go:414] "containerMap: removing container" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container"
Oct 30 19:21:15.317636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317629    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317648    5972 cpu_manager.go:414] "containerMap: removing container" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317654    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317660    5972 cpu_manager.go:414] "containerMap: removing container" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317664    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317677    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317681    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317733    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kserve-container"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317742    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kserve-container"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317747    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9417ec7-5567-4f09-bf4c-a38dc1eba8b3" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317753    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317760    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="78af202f-ea59-4685-8cac-8926806022ce" containerName="kube-rbac-proxy"
Oct 30 19:21:15.317984 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.317767    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3b5362-7d2f-47de-8384-1755f88403f1" containerName="kserve-container"
Oct 30 19:21:15.322226 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.322206    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.372944 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.372915    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 19:21:15.372944 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.372920    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:21:15.421154 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.421129    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 19:21:15.421337 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.421320    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:21:15.422299 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.422282    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"]
Oct 30 19:21:15.443175 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.443157    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.443245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.443183    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.443245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.443219    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.443364 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.443346    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.443407 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.443394    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tw2g\" (UniqueName: \"kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.473840 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.473820    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:21:15.473928 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.473911    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:21:15.473965 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.473922    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:21:15.544689 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.544663    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.544815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.544707    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tw2g\" (UniqueName: \"kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.544815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.544734    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.544815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.544753    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.544815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.544782    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.545297 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.545276    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.545430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.545413    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.545479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.545464    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.547256 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.547239    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.604853 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.604787    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tw2g\" (UniqueName: \"kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g\") pod \"isvc-canary-promote-predictor-58b554b449-7q2tg\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:15.638285 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:15.638250    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:16.063041 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:16.063009    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"]
Oct 30 19:21:16.065806 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:21:16.065779    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2ed420_45f7_49af_bfee_26e102a268e8.slice/crio-81641858e8866d7510f1276282e533543a06d7027621eed8054e89bb43694559 WatchSource:0}: Error finding container 81641858e8866d7510f1276282e533543a06d7027621eed8054e89bb43694559: Status 404 returned error can't find the container with id 81641858e8866d7510f1276282e533543a06d7027621eed8054e89bb43694559
Oct 30 19:21:16.548126 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:16.548096    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerStarted","Data":"c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961"}
Oct 30 19:21:16.548508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:16.548150    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerStarted","Data":"81641858e8866d7510f1276282e533543a06d7027621eed8054e89bb43694559"}
Oct 30 19:21:20.564868 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:20.564834    5972 generic.go:356] "Generic (PLEG): container finished" podID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerID="c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961" exitCode=0
Oct 30 19:21:20.564868 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:20.564870    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerDied","Data":"c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961"}
Oct 30 19:21:21.571045 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:21.571014    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerStarted","Data":"66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb"}
Oct 30 19:21:21.571435 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:21.571072    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerStarted","Data":"45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435"}
Oct 30 19:21:21.571435 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:21.571273    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:21.689874 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:21.689825    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podStartSLOduration=2.699744797 podStartE2EDuration="6.68981166s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990066863s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:21.689360828 +0000 UTC m=+2531.481328127" watchObservedRunningTime="2026-10-30 19:21:21.68981166 +0000 UTC m=+2531.481778945"
Oct 30 19:21:22.574822 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:22.574789    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:22.576042 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:22.576014    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:21:22.868964 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:21:22.868886    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:21:23.578421 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:23.578383    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:21:28.582973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:28.582942    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:21:28.583419 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:28.583355    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:21:38.583504 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:38.583466    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:21:48.583976 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:48.583938    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:21:52.869201 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:21:52.869166    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:21:57.949225 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:57.949195    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"]
Oct 30 19:21:57.952943 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:57.952923    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.003996 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.003970    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:21:58.003996 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.003972    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:21:58.051642 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.051621    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:21:58.051642 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.051623    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:21:58.052034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.051992    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:21:58.052034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.052004    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 30 19:21:58.052215 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.052054    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:21:58.053089 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.053070    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"]
Oct 30 19:21:58.099446 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.099420    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.099563 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.099543    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.099651 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.099631    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.099696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.099658    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.099749 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.099690    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkzj\" (UniqueName: \"kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.200969 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.200902    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.200969 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.200943    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkzj\" (UniqueName: \"kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.200969 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.200964    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.201186 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.201024    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.201186 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.201043    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.201474 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.201443    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.201896 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.201876    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.201957 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.201876    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.203633 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.203612    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.260022 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.260002    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkzj\" (UniqueName: \"kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj\") pod \"isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.270017 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.269997    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:21:58.584385 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.584308    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:21:58.686781 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.686735    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"]
Oct 30 19:21:58.689663 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:21:58.689637    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cddadc_03b2_4fda_b79a_ba80127abb73.slice/crio-a15fb6178c90c50402b2f3de8aec445a2211283e968cc2b37103873efcdb7eb1 WatchSource:0}: Error finding container a15fb6178c90c50402b2f3de8aec445a2211283e968cc2b37103873efcdb7eb1: Status 404 returned error can't find the container with id a15fb6178c90c50402b2f3de8aec445a2211283e968cc2b37103873efcdb7eb1
Oct 30 19:21:58.718086 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:58.718063    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerStarted","Data":"a15fb6178c90c50402b2f3de8aec445a2211283e968cc2b37103873efcdb7eb1"}
Oct 30 19:21:59.723730 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:21:59.723690    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerStarted","Data":"92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428"}
Oct 30 19:22:02.736946 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:02.736910    5972 generic.go:356] "Generic (PLEG): container finished" podID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerID="92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428" exitCode=0
Oct 30 19:22:02.737318 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:02.736985    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerDied","Data":"92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428"}
Oct 30 19:22:03.743275 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:03.743241    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerStarted","Data":"78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df"}
Oct 30 19:22:03.743772 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:03.743302    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerStarted","Data":"4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80"}
Oct 30 19:22:03.743772 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:03.743575    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:22:03.743772 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:03.743640    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:22:03.744894 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:03.744871    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:03.886601 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:03.886540    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podStartSLOduration=3.225854724 podStartE2EDuration="6.886528486s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.660673762s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:03.885424687 +0000 UTC m=+2573.677391975" watchObservedRunningTime="2026-10-30 19:22:03.886528486 +0000 UTC m=+2573.678495768"
Oct 30 19:22:04.747449 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:04.747402    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:08.583409 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:08.583369    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:22:09.751863 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:09.751832    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:22:09.752350 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:09.752322    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:18.584123 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:18.584083    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:22:19.753237 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:19.753202    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:22.869241 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:22:22.869204    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:22:28.583703 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:28.583662    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 19:22:29.752375 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:29.752332    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:38.584398 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:38.584368    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:22:39.752275 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:39.752232    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:40.592429 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:40.592402    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"]
Oct 30 19:22:40.592684 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:40.592636    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container" containerID="cri-o://45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435" gracePeriod=30
Oct 30 19:22:40.592745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:40.592665    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kube-rbac-proxy" containerID="cri-o://66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb" gracePeriod=30
Oct 30 19:22:40.887533 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:40.887453    5972 generic.go:356] "Generic (PLEG): container finished" podID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerID="66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb" exitCode=0
Oct 30 19:22:40.887915 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:40.887526    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerDied","Data":"66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb"}
Oct 30 19:22:43.579135 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:43.579097    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Oct 30 19:22:45.336480 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.336454    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:22:45.488860 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.488830    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls\") pod \"5c2ed420-45f7-49af-bfee-26e102a268e8\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") "
Oct 30 19:22:45.489044 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.488869    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g\" (UniqueName: \"kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g\") pod \"5c2ed420-45f7-49af-bfee-26e102a268e8\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") "
Oct 30 19:22:45.489044 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.488926    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config\") pod \"5c2ed420-45f7-49af-bfee-26e102a268e8\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") "
Oct 30 19:22:45.489044 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.488999    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location\") pod \"5c2ed420-45f7-49af-bfee-26e102a268e8\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") "
Oct 30 19:22:45.489044 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.489029    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert\") pod \"5c2ed420-45f7-49af-bfee-26e102a268e8\" (UID: \"5c2ed420-45f7-49af-bfee-26e102a268e8\") "
Oct 30 19:22:45.489348 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.489324    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location" pod "5c2ed420-45f7-49af-bfee-26e102a268e8" (UID: "5c2ed420-45f7-49af-bfee-26e102a268e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:45.489414 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.489388    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config" pod "5c2ed420-45f7-49af-bfee-26e102a268e8" (UID: "5c2ed420-45f7-49af-bfee-26e102a268e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:45.489472 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.489433    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert" pod "5c2ed420-45f7-49af-bfee-26e102a268e8" (UID: "5c2ed420-45f7-49af-bfee-26e102a268e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:45.491097 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.491078    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls" pod "5c2ed420-45f7-49af-bfee-26e102a268e8" (UID: "5c2ed420-45f7-49af-bfee-26e102a268e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:45.491184 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.491139    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g" pod "5c2ed420-45f7-49af-bfee-26e102a268e8" (UID: "5c2ed420-45f7-49af-bfee-26e102a268e8"). InnerVolumeSpecName "kube-api-access-9tw2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:45.590604 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.590556    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2ed420-45f7-49af-bfee-26e102a268e8-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.590784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.590614    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.590784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.590629    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2ed420-45f7-49af-bfee-26e102a268e8-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.590784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.590642    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tw2g\" (UniqueName: \"kubernetes.io/projected/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-api-access-9tw2g\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.590784 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.590654    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2ed420-45f7-49af-bfee-26e102a268e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.707590 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707541    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"]
Oct 30 19:22:45.707902 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707889    5972 cpu_manager.go:414] "containerMap: removing container" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kube-rbac-proxy"
Oct 30 19:22:45.707945 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707905    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kube-rbac-proxy"
Oct 30 19:22:45.707945 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707921    5972 cpu_manager.go:414] "containerMap: removing container" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container"
Oct 30 19:22:45.707945 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707927    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container"
Oct 30 19:22:45.707945 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707937    5972 cpu_manager.go:414] "containerMap: removing container" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="storage-initializer"
Oct 30 19:22:45.707945 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.707942    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="storage-initializer"
Oct 30 19:22:45.708092 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.708010    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kube-rbac-proxy"
Oct 30 19:22:45.708092 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.708019    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerName="kserve-container"
Oct 30 19:22:45.711288 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.711270    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.762408 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.762350    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 19:22:45.767285 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.767263    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"]
Oct 30 19:22:45.792696 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.792669    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.792815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.792727    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l84q\" (UniqueName: \"kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.792815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.792747    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.792894 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.792840    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.792894 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.792867    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.893899 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.893856    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l84q\" (UniqueName: \"kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.893899 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.893904    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.894126 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.893940    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.894126 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.893982    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.894126 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.894033    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.894500 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.894472    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.894695 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.894674    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.894695 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.894687    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.896451 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.896435    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:45.909228 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.909206    5972 generic.go:356] "Generic (PLEG): container finished" podID="5c2ed420-45f7-49af-bfee-26e102a268e8" containerID="45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435" exitCode=0
Oct 30 19:22:45.909320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.909254    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerDied","Data":"45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435"}
Oct 30 19:22:45.909320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.909267    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"
Oct 30 19:22:45.909320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.909297    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg" event={"ID":"5c2ed420-45f7-49af-bfee-26e102a268e8","Type":"ContainerDied","Data":"81641858e8866d7510f1276282e533543a06d7027621eed8054e89bb43694559"}
Oct 30 19:22:45.909414 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.909323    5972 scope.go:122] "RemoveContainer" containerID="66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb"
Oct 30 19:22:45.917942 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.917928    5972 scope.go:122] "RemoveContainer" containerID="45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435"
Oct 30 19:22:45.925273 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.925260    5972 scope.go:122] "RemoveContainer" containerID="c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961"
Oct 30 19:22:45.932454 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.932435    5972 scope.go:122] "RemoveContainer" containerID="66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb"
Oct 30 19:22:45.932721 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:22:45.932704    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb\": container with ID starting with 66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb not found: ID does not exist" containerID="66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb"
Oct 30 19:22:45.932778 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.932729    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb"} err="failed to get container status \"66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb\": rpc error: code = NotFound desc = could not find container \"66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb\": container with ID starting with 66d60b7cb72fb131e5374589afcc029dc0dd401921b63ff9ea82c90563d1f0bb not found: ID does not exist"
Oct 30 19:22:45.932778 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.932747    5972 scope.go:122] "RemoveContainer" containerID="45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435"
Oct 30 19:22:45.932981 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:22:45.932964    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435\": container with ID starting with 45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435 not found: ID does not exist" containerID="45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435"
Oct 30 19:22:45.933017 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.932987    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435"} err="failed to get container status \"45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435\": rpc error: code = NotFound desc = could not find container \"45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435\": container with ID starting with 45d0c78bf1e0e57762f407eade9467fa8be2e2f5ff39f151298bfbd391416435 not found: ID does not exist"
Oct 30 19:22:45.933017 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.933002    5972 scope.go:122] "RemoveContainer" containerID="c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961"
Oct 30 19:22:45.933206 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:22:45.933191    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961\": container with ID starting with c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961 not found: ID does not exist" containerID="c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961"
Oct 30 19:22:45.933245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.933210    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961"} err="failed to get container status \"c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961\": rpc error: code = NotFound desc = could not find container \"c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961\": container with ID starting with c9f4d1dc9d64edb0c391f543c745e64952c17ee1dcaa7267e995684ce9a5a961 not found: ID does not exist"
Oct 30 19:22:45.953516 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:45.953499    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l84q\" (UniqueName: \"kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q\") pod \"isvc-canary-promote-v2-predictor-76656f7564-dlbnw\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:46.028728 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.028660    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:46.070919 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.070889    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"]
Oct 30 19:22:46.076100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.076063    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-7q2tg"]
Oct 30 19:22:46.339446 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.339371    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2ed420-45f7-49af-bfee-26e102a268e8" path="/var/lib/kubelet/pods/5c2ed420-45f7-49af-bfee-26e102a268e8/volumes"
Oct 30 19:22:46.443965 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.443683    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"]
Oct 30 19:22:46.446506 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:22:46.446476    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e9d991_1e29_4f41_872d_528651ef5d09.slice/crio-49639729a086e9f316104c6e80581cf7ecb2e301f660f781c7c025c6b540e551 WatchSource:0}: Error finding container 49639729a086e9f316104c6e80581cf7ecb2e301f660f781c7c025c6b540e551: Status 404 returned error can't find the container with id 49639729a086e9f316104c6e80581cf7ecb2e301f660f781c7c025c6b540e551
Oct 30 19:22:46.916084 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.916046    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerStarted","Data":"e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56"}
Oct 30 19:22:46.916264 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:46.916111    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerStarted","Data":"49639729a086e9f316104c6e80581cf7ecb2e301f660f781c7c025c6b540e551"}
Oct 30 19:22:49.752807 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:49.752725    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:22:50.934381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:50.934346    5972 generic.go:356] "Generic (PLEG): container finished" podID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerID="e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56" exitCode=0
Oct 30 19:22:50.934781 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:50.934420    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerDied","Data":"e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56"}
Oct 30 19:22:51.940350 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:51.940320    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerStarted","Data":"61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7"}
Oct 30 19:22:51.940755 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:51.940383    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerStarted","Data":"15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6"}
Oct 30 19:22:51.940755 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:51.940559    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:52.057719 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:52.057667    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podStartSLOduration=3.33998143 podStartE2EDuration="7.057652786s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717671356s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.05728016 +0000 UTC m=+2621.849247460" watchObservedRunningTime="2026-10-30 19:22:52.057652786 +0000 UTC m=+2621.849620064"
Oct 30 19:22:52.869800 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:22:52.869768    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:22:52.945145 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:52.945111    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:52.946483 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:52.946454    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:22:53.949517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:53.949480    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:22:58.954495 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:58.954466    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:22:58.954997 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:58.954971    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:22:59.752703 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:22:59.752664    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:23:08.955243 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:08.955204    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:23:09.752785 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:09.752759    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:23:18.631599 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:18.631536    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"]
Oct 30 19:23:18.632103 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:18.631867    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kube-rbac-proxy" containerID="cri-o://78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df" gracePeriod=30
Oct 30 19:23:18.632103 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:18.631847    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" containerID="cri-o://4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80" gracePeriod=30
Oct 30 19:23:18.955360 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:18.955318    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:23:19.052830 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:19.052800    5972 generic.go:356] "Generic (PLEG): container finished" podID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerID="78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df" exitCode=0
Oct 30 19:23:19.052996 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:19.052870    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerDied","Data":"78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df"}
Oct 30 19:23:19.748065 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:19.748025    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Oct 30 19:23:19.752396 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:19.752371    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 19:23:22.870393 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:23:22.870353    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:23:23.368420 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.368400    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:23:23.421460 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421427    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config\") pod \"d5cddadc-03b2-4fda-b79a-ba80127abb73\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") "
Oct 30 19:23:23.421460 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421462    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj\" (UniqueName: \"kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj\") pod \"d5cddadc-03b2-4fda-b79a-ba80127abb73\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") "
Oct 30 19:23:23.421743 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421486    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location\") pod \"d5cddadc-03b2-4fda-b79a-ba80127abb73\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") "
Oct 30 19:23:23.421743 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421524    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert\") pod \"d5cddadc-03b2-4fda-b79a-ba80127abb73\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") "
Oct 30 19:23:23.421743 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421639    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls\") pod \"d5cddadc-03b2-4fda-b79a-ba80127abb73\" (UID: \"d5cddadc-03b2-4fda-b79a-ba80127abb73\") "
Oct 30 19:23:23.421904 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421862    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config" pod "d5cddadc-03b2-4fda-b79a-ba80127abb73" (UID: "d5cddadc-03b2-4fda-b79a-ba80127abb73"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:23.421904 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421867    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location" pod "d5cddadc-03b2-4fda-b79a-ba80127abb73" (UID: "d5cddadc-03b2-4fda-b79a-ba80127abb73"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:23.421983 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.421941    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert" pod "d5cddadc-03b2-4fda-b79a-ba80127abb73" (UID: "d5cddadc-03b2-4fda-b79a-ba80127abb73"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:23.423826 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.423796    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj" pod "d5cddadc-03b2-4fda-b79a-ba80127abb73" (UID: "d5cddadc-03b2-4fda-b79a-ba80127abb73"). InnerVolumeSpecName "kube-api-access-vpkzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:23.423903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.423847    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls" pod "d5cddadc-03b2-4fda-b79a-ba80127abb73" (UID: "d5cddadc-03b2-4fda-b79a-ba80127abb73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:23.522536 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.522457    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:23.522536 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.522486    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cddadc-03b2-4fda-b79a-ba80127abb73-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:23.522536 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.522498    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:23.522536 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.522507    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkzj\" (UniqueName: \"kubernetes.io/projected/d5cddadc-03b2-4fda-b79a-ba80127abb73-kube-api-access-vpkzj\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:23.522536 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:23.522515    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cddadc-03b2-4fda-b79a-ba80127abb73-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:24.075101 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.075070    5972 generic.go:356] "Generic (PLEG): container finished" podID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerID="4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80" exitCode=0
Oct 30 19:23:24.075101 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.075106    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerDied","Data":"4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80"}
Oct 30 19:23:24.075855 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.075146    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"
Oct 30 19:23:24.075855 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.075162    5972 scope.go:122] "RemoveContainer" containerID="78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df"
Oct 30 19:23:24.075855 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.075152    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw" event={"ID":"d5cddadc-03b2-4fda-b79a-ba80127abb73","Type":"ContainerDied","Data":"a15fb6178c90c50402b2f3de8aec445a2211283e968cc2b37103873efcdb7eb1"}
Oct 30 19:23:24.083530 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.083508    5972 scope.go:122] "RemoveContainer" containerID="4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80"
Oct 30 19:23:24.091051 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.091033    5972 scope.go:122] "RemoveContainer" containerID="92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428"
Oct 30 19:23:24.099106 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.099078    5972 scope.go:122] "RemoveContainer" containerID="78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df"
Oct 30 19:23:24.099401 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:23:24.099380    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df\": container with ID starting with 78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df not found: ID does not exist" containerID="78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df"
Oct 30 19:23:24.099467 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.099410    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df"} err="failed to get container status \"78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df\": rpc error: code = NotFound desc = could not find container \"78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df\": container with ID starting with 78669358e397035348a564daad528e08551e68d4155c79d961c1bac50adb66df not found: ID does not exist"
Oct 30 19:23:24.099467 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.099437    5972 scope.go:122] "RemoveContainer" containerID="4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80"
Oct 30 19:23:24.099740 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:23:24.099724    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80\": container with ID starting with 4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80 not found: ID does not exist" containerID="4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80"
Oct 30 19:23:24.099814 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.099749    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80"} err="failed to get container status \"4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80\": rpc error: code = NotFound desc = could not find container \"4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80\": container with ID starting with 4f2aaa481c4fccbd6553b086b81313dc9d41bd8691bd327b22b2df320a712b80 not found: ID does not exist"
Oct 30 19:23:24.099814 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.099770    5972 scope.go:122] "RemoveContainer" containerID="92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428"
Oct 30 19:23:24.100026 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:23:24.100010    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428\": container with ID starting with 92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428 not found: ID does not exist" containerID="92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428"
Oct 30 19:23:24.100079 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.100033    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428"} err="failed to get container status \"92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428\": rpc error: code = NotFound desc = could not find container \"92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428\": container with ID starting with 92d95d9917f2fff9775fb9dc89ee1d7a700b8128de270efeb6d0a528ae76f428 not found: ID does not exist"
Oct 30 19:23:24.281968 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.281932    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"]
Oct 30 19:23:24.329811 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.329723    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-runtime-predictor-86b7d46b9b-qjcmw"]
Oct 30 19:23:24.339038 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:24.339001    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" path="/var/lib/kubelet/pods/d5cddadc-03b2-4fda-b79a-ba80127abb73/volumes"
Oct 30 19:23:28.955476 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:28.955434    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:23:38.955630 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:38.955569    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:23:48.955669 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:48.955629    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:23:52.869953 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:23:52.869905    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:23:58.954964 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:23:58.954925    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:24:08.955750 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:08.955717    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:24:10.624541 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:10.624514    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:24:10.648162 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:10.648136    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:24:15.444436 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:15.444405    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"]
Oct 30 19:24:15.444892 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:15.444654    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" containerID="cri-o://15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6" gracePeriod=30
Oct 30 19:24:15.444892 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:15.444698    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kube-rbac-proxy" containerID="cri-o://61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7" gracePeriod=30
Oct 30 19:24:16.285128 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:16.285094    5972 generic.go:356] "Generic (PLEG): container finished" podID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerID="61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7" exitCode=0
Oct 30 19:24:16.285306 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:16.285167    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerDied","Data":"61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7"}
Oct 30 19:24:18.950145 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:18.950066    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 19:24:18.955405 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:18.955370    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 19:24:20.289977 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.289955    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:24:20.304014 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.303986    5972 generic.go:356] "Generic (PLEG): container finished" podID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerID="15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6" exitCode=0
Oct 30 19:24:20.304148 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.304047    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"
Oct 30 19:24:20.304148 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.304052    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerDied","Data":"15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6"}
Oct 30 19:24:20.304148 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.304108    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw" event={"ID":"f8e9d991-1e29-4f41-872d-528651ef5d09","Type":"ContainerDied","Data":"49639729a086e9f316104c6e80581cf7ecb2e301f660f781c7c025c6b540e551"}
Oct 30 19:24:20.304309 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.304182    5972 scope.go:122] "RemoveContainer" containerID="61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7"
Oct 30 19:24:20.313417 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.313396    5972 scope.go:122] "RemoveContainer" containerID="15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6"
Oct 30 19:24:20.320778 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.320762    5972 scope.go:122] "RemoveContainer" containerID="e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56"
Oct 30 19:24:20.327757 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.327744    5972 scope.go:122] "RemoveContainer" containerID="61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7"
Oct 30 19:24:20.327991 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:24:20.327974    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7\": container with ID starting with 61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7 not found: ID does not exist" containerID="61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7"
Oct 30 19:24:20.328035 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.328000    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7"} err="failed to get container status \"61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7\": rpc error: code = NotFound desc = could not find container \"61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7\": container with ID starting with 61f6eaa9d9721ebfb809979a28ddecf19c170bf2108bdfca23b96c0cbddd10f7 not found: ID does not exist"
Oct 30 19:24:20.328035 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.328015    5972 scope.go:122] "RemoveContainer" containerID="15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6"
Oct 30 19:24:20.328230 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:24:20.328214    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6\": container with ID starting with 15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6 not found: ID does not exist" containerID="15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6"
Oct 30 19:24:20.328270 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.328233    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6"} err="failed to get container status \"15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6\": rpc error: code = NotFound desc = could not find container \"15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6\": container with ID starting with 15e33a629e9ac2e3beab8f7e419904e8c1d9065a69392d350a091ea2eeb211f6 not found: ID does not exist"
Oct 30 19:24:20.328270 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.328245    5972 scope.go:122] "RemoveContainer" containerID="e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56"
Oct 30 19:24:20.328428 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:24:20.328412    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56\": container with ID starting with e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56 not found: ID does not exist" containerID="e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56"
Oct 30 19:24:20.328463 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.328432    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56"} err="failed to get container status \"e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56\": rpc error: code = NotFound desc = could not find container \"e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56\": container with ID starting with e8e3d32ac1102d9ad02b9321329e6b97717e2f5850176efd0af22bcc589dcd56 not found: ID does not exist"
Oct 30 19:24:20.405705 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.405639    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config\") pod \"f8e9d991-1e29-4f41-872d-528651ef5d09\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") "
Oct 30 19:24:20.405705 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.405699    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location\") pod \"f8e9d991-1e29-4f41-872d-528651ef5d09\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") "
Oct 30 19:24:20.405903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.405749    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls\") pod \"f8e9d991-1e29-4f41-872d-528651ef5d09\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") "
Oct 30 19:24:20.405903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.405799    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert\") pod \"f8e9d991-1e29-4f41-872d-528651ef5d09\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") "
Oct 30 19:24:20.405903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.405869    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q\" (UniqueName: \"kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q\") pod \"f8e9d991-1e29-4f41-872d-528651ef5d09\" (UID: \"f8e9d991-1e29-4f41-872d-528651ef5d09\") "
Oct 30 19:24:20.406064 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.406028    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config" pod "f8e9d991-1e29-4f41-872d-528651ef5d09" (UID: "f8e9d991-1e29-4f41-872d-528651ef5d09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:20.406158 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.406046    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location" pod "f8e9d991-1e29-4f41-872d-528651ef5d09" (UID: "f8e9d991-1e29-4f41-872d-528651ef5d09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:20.406263 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.406236    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert" pod "f8e9d991-1e29-4f41-872d-528651ef5d09" (UID: "f8e9d991-1e29-4f41-872d-528651ef5d09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:20.407919 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.407898    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls" pod "f8e9d991-1e29-4f41-872d-528651ef5d09" (UID: "f8e9d991-1e29-4f41-872d-528651ef5d09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:20.408009 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.407993    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q" pod "f8e9d991-1e29-4f41-872d-528651ef5d09" (UID: "f8e9d991-1e29-4f41-872d-528651ef5d09"). InnerVolumeSpecName "kube-api-access-7l84q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:20.506958 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.506922    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.506958 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.506959    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e9d991-1e29-4f41-872d-528651ef5d09-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.507152 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.506974    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e9d991-1e29-4f41-872d-528651ef5d09-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.507152 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.506991    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e9d991-1e29-4f41-872d-528651ef5d09-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.507152 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.507007    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l84q\" (UniqueName: \"kubernetes.io/projected/f8e9d991-1e29-4f41-872d-528651ef5d09-kube-api-access-7l84q\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.720179 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.720140    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"]
Oct 30 19:24:20.724516 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:20.724492    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-dlbnw"]
Oct 30 19:24:22.337795 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:22.337764    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" path="/var/lib/kubelet/pods/f8e9d991-1e29-4f41-872d-528651ef5d09/volumes"
Oct 30 19:24:22.869431 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:24:22.869387    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:24:25.084825 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.084787    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"]
Oct 30 19:24:25.085280 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085250    5972 cpu_manager.go:414] "containerMap: removing container" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="storage-initializer"
Oct 30 19:24:25.085280 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085268    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="storage-initializer"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085306    5972 cpu_manager.go:414] "containerMap: removing container" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="storage-initializer"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085314    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="storage-initializer"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085340    5972 cpu_manager.go:414] "containerMap: removing container" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kube-rbac-proxy"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085347    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kube-rbac-proxy"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085362    5972 cpu_manager.go:414] "containerMap: removing container" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085369    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085382    5972 cpu_manager.go:414] "containerMap: removing container" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085389    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085399    5972 cpu_manager.go:414] "containerMap: removing container" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kube-rbac-proxy"
Oct 30 19:24:25.085403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085406    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kube-rbac-proxy"
Oct 30 19:24:25.085902 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085472    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kube-rbac-proxy"
Oct 30 19:24:25.085902 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085485    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kserve-container"
Oct 30 19:24:25.085902 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085495    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cddadc-03b2-4fda-b79a-ba80127abb73" containerName="kserve-container"
Oct 30 19:24:25.085902 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.085504    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e9d991-1e29-4f41-872d-528651ef5d09" containerName="kube-rbac-proxy"
Oct 30 19:24:25.090365 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.090343    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.144025 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.143992    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:24:25.144025 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.144014    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:24:25.144329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.144030    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 19:24:25.144329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.144266    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:24:25.144329 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.144291    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:24:25.191667 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.191651    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:24:25.192225 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.192189    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 19:24:25.192761 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.192742    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"]
Oct 30 19:24:25.248672 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.248649    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.248760 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.248691    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhgd\" (UniqueName: \"kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.248760 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.248719    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.248888 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.248869    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.248935 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.248899    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.350307 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.350276    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.350464 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.350332    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.350464 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.350362    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.350464 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.350409    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.350464 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.350450    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhgd\" (UniqueName: \"kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.350824 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.350807    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.351073 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.351054    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.351157 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.351139    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.353035 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.353014    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.410417 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.410397    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhgd\" (UniqueName: \"kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-l9sgk\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:25.706753 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:25.706723    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:26.120797 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:26.120722    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"]
Oct 30 19:24:26.123994 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:24:26.123961    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cee8a2d_422b_4006_9dc8_d872507d4c74.slice/crio-e645a01e5d20f57c13d3a8d8d70064ff0f43bca2a43469c79dc895dc11a1fd11 WatchSource:0}: Error finding container e645a01e5d20f57c13d3a8d8d70064ff0f43bca2a43469c79dc895dc11a1fd11: Status 404 returned error can't find the container with id e645a01e5d20f57c13d3a8d8d70064ff0f43bca2a43469c79dc895dc11a1fd11
Oct 30 19:24:26.331083 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:26.331051    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerStarted","Data":"1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8"}
Oct 30 19:24:26.331270 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:26.331105    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerStarted","Data":"e645a01e5d20f57c13d3a8d8d70064ff0f43bca2a43469c79dc895dc11a1fd11"}
Oct 30 19:24:30.346601 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:30.346554    5972 generic.go:356] "Generic (PLEG): container finished" podID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerID="1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8" exitCode=0
Oct 30 19:24:30.347086 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:30.346637    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerDied","Data":"1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8"}
Oct 30 19:24:31.353075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:31.353034    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerStarted","Data":"0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2"}
Oct 30 19:24:31.353463 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:31.353150    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerStarted","Data":"7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e"}
Oct 30 19:24:31.353463 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:31.353430    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:31.353550 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:31.353486    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:31.354839 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:31.354813    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:24:31.565242 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:31.565196    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podStartSLOduration=2.839587811 podStartE2EDuration="6.565182261s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.72559445s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:31.517010128 +0000 UTC m=+2721.308977416" watchObservedRunningTime="2026-10-30 19:24:31.565182261 +0000 UTC m=+2721.357149543"
Oct 30 19:24:32.357301 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:32.357256    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:24:37.361691 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:37.361663    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:24:37.362343 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:37.362311    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:24:47.362442 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:47.362396    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:24:52.870171 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:24:52.870125    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:24:57.363121 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:24:57.363079    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:25:07.362320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:25:07.362276    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:25:17.362796 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:25:17.362755    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:25:22.869629 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:25:22.869576    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:25:27.362849 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:25:27.362806    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:25:37.362374 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:25:37.362336    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:25:47.334488 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:25:47.334458    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:25:52.870218 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:25:52.870133    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:26:22.869334 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:26:22.869294    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:26:52.869462 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:26:52.869424    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:27:22.870071 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:27:22.869988    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:27:35.033007 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:35.032966    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"]
Oct 30 19:27:35.033489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:35.033262    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" containerID="cri-o://7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e" gracePeriod=30
Oct 30 19:27:35.033489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:35.033291    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kube-rbac-proxy" containerID="cri-o://0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2" gracePeriod=30
Oct 30 19:27:36.072432 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:36.072398    5972 generic.go:356] "Generic (PLEG): container finished" podID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerID="0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2" exitCode=0
Oct 30 19:27:36.072818 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:36.072467    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerDied","Data":"0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2"}
Oct 30 19:27:37.333921 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:37.333878    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 19:27:37.357541 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:37.357512    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Oct 30 19:27:39.371066 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.371044    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:27:39.522633 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.522544    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert\") pod \"4cee8a2d-422b-4006-9dc8-d872507d4c74\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") "
Oct 30 19:27:39.522633 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.522596    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd\" (UniqueName: \"kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd\") pod \"4cee8a2d-422b-4006-9dc8-d872507d4c74\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") "
Oct 30 19:27:39.522633 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.522627    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls\") pod \"4cee8a2d-422b-4006-9dc8-d872507d4c74\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") "
Oct 30 19:27:39.522848 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.522645    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config\") pod \"4cee8a2d-422b-4006-9dc8-d872507d4c74\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") "
Oct 30 19:27:39.522848 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.522673    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location\") pod \"4cee8a2d-422b-4006-9dc8-d872507d4c74\" (UID: \"4cee8a2d-422b-4006-9dc8-d872507d4c74\") "
Oct 30 19:27:39.522927 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.522891    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert" pod "4cee8a2d-422b-4006-9dc8-d872507d4c74" (UID: "4cee8a2d-422b-4006-9dc8-d872507d4c74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:39.523048 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.523027    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config" pod "4cee8a2d-422b-4006-9dc8-d872507d4c74" (UID: "4cee8a2d-422b-4006-9dc8-d872507d4c74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:39.523107 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.523057    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location" pod "4cee8a2d-422b-4006-9dc8-d872507d4c74" (UID: "4cee8a2d-422b-4006-9dc8-d872507d4c74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:39.524907 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.524889    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls" pod "4cee8a2d-422b-4006-9dc8-d872507d4c74" (UID: "4cee8a2d-422b-4006-9dc8-d872507d4c74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:39.524987 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.524948    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd" pod "4cee8a2d-422b-4006-9dc8-d872507d4c74" (UID: "4cee8a2d-422b-4006-9dc8-d872507d4c74"). InnerVolumeSpecName "kube-api-access-bqhgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:39.624058 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.624031    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.624058 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.624055    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhgd\" (UniqueName: \"kubernetes.io/projected/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-api-access-bqhgd\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.624238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.624065    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cee8a2d-422b-4006-9dc8-d872507d4c74-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.624238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.624076    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cee8a2d-422b-4006-9dc8-d872507d4c74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.624238 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:39.624086    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cee8a2d-422b-4006-9dc8-d872507d4c74-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.089666 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.089628    5972 generic.go:356] "Generic (PLEG): container finished" podID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerID="7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e" exitCode=0
Oct 30 19:27:40.089871 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.089695    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"
Oct 30 19:27:40.089871 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.089721    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerDied","Data":"7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e"}
Oct 30 19:27:40.089871 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.089774    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk" event={"ID":"4cee8a2d-422b-4006-9dc8-d872507d4c74","Type":"ContainerDied","Data":"e645a01e5d20f57c13d3a8d8d70064ff0f43bca2a43469c79dc895dc11a1fd11"}
Oct 30 19:27:40.089871 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.089800    5972 scope.go:122] "RemoveContainer" containerID="0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2"
Oct 30 19:27:40.099059 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.099044    5972 scope.go:122] "RemoveContainer" containerID="7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e"
Oct 30 19:27:40.106876 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.106864    5972 scope.go:122] "RemoveContainer" containerID="1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8"
Oct 30 19:27:40.115749 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.115724    5972 scope.go:122] "RemoveContainer" containerID="0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2"
Oct 30 19:27:40.116012 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:27:40.115989    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2\": container with ID starting with 0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2 not found: ID does not exist" containerID="0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2"
Oct 30 19:27:40.116100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.116022    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2"} err="failed to get container status \"0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2\": rpc error: code = NotFound desc = could not find container \"0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2\": container with ID starting with 0052d4224ebdeb03629b97beb55967fc45f35e7aaf112f275237a09422d71ee2 not found: ID does not exist"
Oct 30 19:27:40.116100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.116046    5972 scope.go:122] "RemoveContainer" containerID="7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e"
Oct 30 19:27:40.116274 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:27:40.116253    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e\": container with ID starting with 7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e not found: ID does not exist" containerID="7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e"
Oct 30 19:27:40.116321 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.116281    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e"} err="failed to get container status \"7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e\": rpc error: code = NotFound desc = could not find container \"7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e\": container with ID starting with 7a3e45e752be464c5324c4d2877e2f8d618dcea061681e3a264ba78c8b470e3e not found: ID does not exist"
Oct 30 19:27:40.116321 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.116296    5972 scope.go:122] "RemoveContainer" containerID="1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8"
Oct 30 19:27:40.116540 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:27:40.116524    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8\": container with ID starting with 1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8 not found: ID does not exist" containerID="1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8"
Oct 30 19:27:40.116621 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.116548    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8"} err="failed to get container status \"1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8\": rpc error: code = NotFound desc = could not find container \"1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8\": container with ID starting with 1343d7ec09974a6a2f13730b48a20796d18e370eb4157810c03717e6aa5d3fa8 not found: ID does not exist"
Oct 30 19:27:40.296907 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.296879    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"]
Oct 30 19:27:40.344802 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:40.344740    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-l9sgk"]
Oct 30 19:27:42.338764 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:42.338732    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" path="/var/lib/kubelet/pods/4cee8a2d-422b-4006-9dc8-d872507d4c74/volumes"
Oct 30 19:27:44.755075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755046    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"]
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755360    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kube-rbac-proxy"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755370    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kube-rbac-proxy"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755390    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755396    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755406    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="storage-initializer"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755411    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="storage-initializer"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755459    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kserve-container"
Oct 30 19:27:44.755479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.755474    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cee8a2d-422b-4006-9dc8-d872507d4c74" containerName="kube-rbac-proxy"
Oct 30 19:27:44.760380 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.760360    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.811900 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.811870    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:27:44.812052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.811903    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 19:27:44.812052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.811966    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:27:44.859839 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.859806    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:27:44.859988 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.859839    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:27:44.859988 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.859864    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 19:27:44.859988 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.859963    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:27:44.860729 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.860710    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"]
Oct 30 19:27:44.865304 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.865288    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.865381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.865323    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.865381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.865347    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnzx\" (UniqueName: \"kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.865381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.865367    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.865538 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.865521    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.966863 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.966838    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.966995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.966895    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.966995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.966941    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.966995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.966976    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnzx\" (UniqueName: \"kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.967163 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.967004    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.967479 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.967459    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.967788 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.967631    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.967788 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.967668    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:44.969457 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:44.969440    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:45.026155 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:45.026101    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnzx\" (UniqueName: \"kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx\") pod \"isvc-canary-stop-predictor-69bd766f7-nsvgq\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:45.076851 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:45.076823    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:45.492381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:45.492350    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"]
Oct 30 19:27:45.496551 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:27:45.496524    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762f8357_e91d_4e91_80c7_b33637cc3347.slice/crio-3b23cece2ba2f57a51e685c7a6bcb2f29f7c1fcf23793b7270d6f6e376443405 WatchSource:0}: Error finding container 3b23cece2ba2f57a51e685c7a6bcb2f29f7c1fcf23793b7270d6f6e376443405: Status 404 returned error can't find the container with id 3b23cece2ba2f57a51e685c7a6bcb2f29f7c1fcf23793b7270d6f6e376443405
Oct 30 19:27:46.117404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:46.117371    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerStarted","Data":"bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c"}
Oct 30 19:27:46.117817 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:46.117435    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerStarted","Data":"3b23cece2ba2f57a51e685c7a6bcb2f29f7c1fcf23793b7270d6f6e376443405"}
Oct 30 19:27:50.136995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:50.136962    5972 generic.go:356] "Generic (PLEG): container finished" podID="762f8357-e91d-4e91-80c7-b33637cc3347" containerID="bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c" exitCode=0
Oct 30 19:27:50.137402 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:50.137014    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerDied","Data":"bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c"}
Oct 30 19:27:51.144057 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:51.144020    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerStarted","Data":"f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922"}
Oct 30 19:27:51.144543 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:51.144093    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerStarted","Data":"ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e"}
Oct 30 19:27:51.144543 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:51.144303    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:51.313302 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:51.313255    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podStartSLOduration=3.628279672 podStartE2EDuration="7.313243375s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684963703s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:51.312667481 +0000 UTC m=+2921.104634768" watchObservedRunningTime="2026-10-30 19:27:51.313243375 +0000 UTC m=+2921.105210663"
Oct 30 19:27:52.148139 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:52.148100    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:52.149362 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:52.149337    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:27:52.870109 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:27:52.870077    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:27:53.152139 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:53.152055    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:27:58.157321 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:58.157286    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:27:58.157800 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:27:58.157773    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:08.158256 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:08.158215    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:09.869540 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.869510    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"]
Oct 30 19:28:09.873173 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.873153    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:09.974287 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974248    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:09.974440 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974360    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:28:09.974440 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974399    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:09.974440 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974410    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 30 19:28:09.974440 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974424    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:09.974664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974497    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:28:09.974664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974504    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 30 19:28:09.974664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974497    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:09.974664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974524    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:28:09.974664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974536    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxk9v\" (UniqueName: \"kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:09.974664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974536    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:28:09.974903 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.974889    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:28:09.977071 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:09.977056    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"]
Oct 30 19:28:10.075075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075045    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075075 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075080    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075214    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075248    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxk9v\" (UniqueName: \"kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075320 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075285    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075558 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075534    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075849 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075828    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.075995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.075974    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.077799 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.077783    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.143508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.143443    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxk9v\" (UniqueName: \"kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.190450 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.190430    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:10.609772 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.609749    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"]
Oct 30 19:28:10.612361 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:28:10.612330    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ae90e5_f14a_4bde_a86e_56011807f627.slice/crio-541cdf339c2047ce3e4e032f6c2a6909917f1082b39aa9244c155736d6e6e937 WatchSource:0}: Error finding container 541cdf339c2047ce3e4e032f6c2a6909917f1082b39aa9244c155736d6e6e937: Status 404 returned error can't find the container with id 541cdf339c2047ce3e4e032f6c2a6909917f1082b39aa9244c155736d6e6e937
Oct 30 19:28:10.663851 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:10.663825    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:28:11.227284 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:11.227243    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerStarted","Data":"0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d"}
Oct 30 19:28:11.227710 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:11.227312    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerStarted","Data":"541cdf339c2047ce3e4e032f6c2a6909917f1082b39aa9244c155736d6e6e937"}
Oct 30 19:28:12.232619 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:12.232567    5972 generic.go:356] "Generic (PLEG): container finished" podID="09ae90e5-f14a-4bde-a86e-56011807f627" containerID="0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d" exitCode=0
Oct 30 19:28:12.232619 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:12.232608    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerDied","Data":"0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d"}
Oct 30 19:28:13.239308 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:13.239267    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerStarted","Data":"c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44"}
Oct 30 19:28:13.239824 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:13.239367    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerStarted","Data":"3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105"}
Oct 30 19:28:13.239824 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:13.239500    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:13.239824 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:13.239561    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:13.240776 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:13.240751    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:28:13.413716 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:13.413666    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podStartSLOduration=3.374152204 podStartE2EDuration="4.413652444s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.03950024s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:13.410077197 +0000 UTC m=+2943.202044485" watchObservedRunningTime="2026-10-30 19:28:13.413652444 +0000 UTC m=+2943.205619729"
Oct 30 19:28:14.243987 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:14.243945    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:28:18.158628 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:18.158570    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:19.248283 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:19.248256    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:28:19.248777 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:19.248754    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:28:22.869983 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:28:22.869935    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:28:28.158091 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:28.158053    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:29.249604 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:29.249539    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:28:38.158399 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:38.158362    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:39.248757 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:39.248717    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:28:48.157779 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:48.157730    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:49.249071 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:49.249030    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:28:52.869686 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:28:52.869653    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:28:58.158029 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:58.157986    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 19:28:59.249006 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:28:59.248967    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:29:08.035484 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.035449    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"]
Oct 30 19:29:08.035921 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.035733    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container" containerID="cri-o://ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e" gracePeriod=30
Oct 30 19:29:08.035921 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.035802    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kube-rbac-proxy" containerID="cri-o://f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922" gracePeriod=30
Oct 30 19:29:08.036805 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.036781    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:29:08.152889 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.152855    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Oct 30 19:29:08.459031 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.458998    5972 generic.go:356] "Generic (PLEG): container finished" podID="762f8357-e91d-4e91-80c7-b33637cc3347" containerID="f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922" exitCode=0
Oct 30 19:29:08.459220 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:08.459064    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerDied","Data":"f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922"}
Oct 30 19:29:09.248906 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:09.248866    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:29:10.658780 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:10.658758    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:29:10.677991 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:10.677962    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:29:12.875957 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:12.875930    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:29:13.032166 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032101    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config\") pod \"762f8357-e91d-4e91-80c7-b33637cc3347\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") "
Oct 30 19:29:13.032297 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032187    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert\") pod \"762f8357-e91d-4e91-80c7-b33637cc3347\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") "
Oct 30 19:29:13.032297 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032211    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location\") pod \"762f8357-e91d-4e91-80c7-b33637cc3347\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") "
Oct 30 19:29:13.032297 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032240    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls\" (UniqueName: \"kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls\") pod \"762f8357-e91d-4e91-80c7-b33637cc3347\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") "
Oct 30 19:29:13.032297 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032271    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx\" (UniqueName: \"kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx\") pod \"762f8357-e91d-4e91-80c7-b33637cc3347\" (UID: \"762f8357-e91d-4e91-80c7-b33637cc3347\") "
Oct 30 19:29:13.032520 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032494    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config" pod "762f8357-e91d-4e91-80c7-b33637cc3347" (UID: "762f8357-e91d-4e91-80c7-b33637cc3347"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:13.032609 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032543    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location" pod "762f8357-e91d-4e91-80c7-b33637cc3347" (UID: "762f8357-e91d-4e91-80c7-b33637cc3347"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:13.032609 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.032573    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert" pod "762f8357-e91d-4e91-80c7-b33637cc3347" (UID: "762f8357-e91d-4e91-80c7-b33637cc3347"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:13.034510 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.034483    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx" pod "762f8357-e91d-4e91-80c7-b33637cc3347" (UID: "762f8357-e91d-4e91-80c7-b33637cc3347"). InnerVolumeSpecName "kube-api-access-lbnzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:13.034598 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.034558    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls" pod "762f8357-e91d-4e91-80c7-b33637cc3347" (UID: "762f8357-e91d-4e91-80c7-b33637cc3347"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:13.133092 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.133064    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:13.133092 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.133091    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762f8357-e91d-4e91-80c7-b33637cc3347-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:13.133288 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.133101    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762f8357-e91d-4e91-80c7-b33637cc3347-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:13.133288 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.133110    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762f8357-e91d-4e91-80c7-b33637cc3347-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:13.133288 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.133119    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnzx\" (UniqueName: \"kubernetes.io/projected/762f8357-e91d-4e91-80c7-b33637cc3347-kube-api-access-lbnzx\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:13.480814 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.480779    5972 generic.go:356] "Generic (PLEG): container finished" podID="762f8357-e91d-4e91-80c7-b33637cc3347" containerID="ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e" exitCode=0
Oct 30 19:29:13.480985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.480848    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerDied","Data":"ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e"}
Oct 30 19:29:13.480985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.480893    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq" event={"ID":"762f8357-e91d-4e91-80c7-b33637cc3347","Type":"ContainerDied","Data":"3b23cece2ba2f57a51e685c7a6bcb2f29f7c1fcf23793b7270d6f6e376443405"}
Oct 30 19:29:13.480985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.480918    5972 scope.go:122] "RemoveContainer" containerID="f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922"
Oct 30 19:29:13.481111 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.480848    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"
Oct 30 19:29:13.489620 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.489601    5972 scope.go:122] "RemoveContainer" containerID="ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e"
Oct 30 19:29:13.497215 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.497198    5972 scope.go:122] "RemoveContainer" containerID="bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c"
Oct 30 19:29:13.506107 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.506086    5972 scope.go:122] "RemoveContainer" containerID="f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922"
Oct 30 19:29:13.506360 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:13.506341    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922\": container with ID starting with f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922 not found: ID does not exist" containerID="f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922"
Oct 30 19:29:13.506430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.506377    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922"} err="failed to get container status \"f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922\": rpc error: code = NotFound desc = could not find container \"f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922\": container with ID starting with f83c8e5fa99d4ae676cefa97f12a053eababbe1d18a8ed37c760a747f11f0922 not found: ID does not exist"
Oct 30 19:29:13.506430 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.506403    5972 scope.go:122] "RemoveContainer" containerID="ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e"
Oct 30 19:29:13.506700 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:13.506681    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e\": container with ID starting with ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e not found: ID does not exist" containerID="ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e"
Oct 30 19:29:13.506749 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.506707    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e"} err="failed to get container status \"ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e\": rpc error: code = NotFound desc = could not find container \"ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e\": container with ID starting with ef90246696ad0106cd76f491af00a070e95e2a3f8da01d96ad305478fc3b013e not found: ID does not exist"
Oct 30 19:29:13.506749 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.506723    5972 scope.go:122] "RemoveContainer" containerID="bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c"
Oct 30 19:29:13.506961 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:13.506941    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c\": container with ID starting with bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c not found: ID does not exist" containerID="bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c"
Oct 30 19:29:13.507009 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.506966    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c"} err="failed to get container status \"bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c\": rpc error: code = NotFound desc = could not find container \"bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c\": container with ID starting with bd4f33e96ac7a387aeb30a548368789f4bfe4d2a5411ca79b5baea3c2dbff25c not found: ID does not exist"
Oct 30 19:29:13.639801 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.639772    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"]
Oct 30 19:29:13.687447 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:13.687417    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-nsvgq"]
Oct 30 19:29:14.338985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:14.338945    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" path="/var/lib/kubelet/pods/762f8357-e91d-4e91-80c7-b33637cc3347/volumes"
Oct 30 19:29:19.249429 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:19.249385    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 19:29:22.869996 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:22.869962    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:29:29.249244 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:29.249213    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:29:39.779451 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:39.779418    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"]
Oct 30 19:29:39.779962 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:39.779752    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kube-rbac-proxy" containerID="cri-o://c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44" gracePeriod=30
Oct 30 19:29:39.779962 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:39.779746    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container" containerID="cri-o://3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105" gracePeriod=30
Oct 30 19:29:40.587658 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:40.587624    5972 generic.go:356] "Generic (PLEG): container finished" podID="09ae90e5-f14a-4bde-a86e-56011807f627" containerID="c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44" exitCode=0
Oct 30 19:29:40.587824 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:40.587700    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerDied","Data":"c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44"}
Oct 30 19:29:44.244715 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.244669    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Oct 30 19:29:44.523301 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.523280    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:29:44.583884 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.583860    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config\") pod \"09ae90e5-f14a-4bde-a86e-56011807f627\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") "
Oct 30 19:29:44.584015 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.583896    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v\" (UniqueName: \"kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v\") pod \"09ae90e5-f14a-4bde-a86e-56011807f627\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") "
Oct 30 19:29:44.584015 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.583933    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert\") pod \"09ae90e5-f14a-4bde-a86e-56011807f627\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") "
Oct 30 19:29:44.584015 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.583993    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls\") pod \"09ae90e5-f14a-4bde-a86e-56011807f627\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") "
Oct 30 19:29:44.584191 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.584016    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location\") pod \"09ae90e5-f14a-4bde-a86e-56011807f627\" (UID: \"09ae90e5-f14a-4bde-a86e-56011807f627\") "
Oct 30 19:29:44.584335 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.584300    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert" pod "09ae90e5-f14a-4bde-a86e-56011807f627" (UID: "09ae90e5-f14a-4bde-a86e-56011807f627"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:44.584335 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.584317    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config" pod "09ae90e5-f14a-4bde-a86e-56011807f627" (UID: "09ae90e5-f14a-4bde-a86e-56011807f627"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:44.584485 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.584366    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location" pod "09ae90e5-f14a-4bde-a86e-56011807f627" (UID: "09ae90e5-f14a-4bde-a86e-56011807f627"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:44.586159 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.586138    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls" pod "09ae90e5-f14a-4bde-a86e-56011807f627" (UID: "09ae90e5-f14a-4bde-a86e-56011807f627"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:44.586240 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.586194    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v" pod "09ae90e5-f14a-4bde-a86e-56011807f627" (UID: "09ae90e5-f14a-4bde-a86e-56011807f627"). InnerVolumeSpecName "kube-api-access-qxk9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:44.604606 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.604560    5972 generic.go:356] "Generic (PLEG): container finished" podID="09ae90e5-f14a-4bde-a86e-56011807f627" containerID="3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105" exitCode=0
Oct 30 19:29:44.604702 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.604631    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"
Oct 30 19:29:44.604702 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.604637    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerDied","Data":"3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105"}
Oct 30 19:29:44.604702 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.604695    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth" event={"ID":"09ae90e5-f14a-4bde-a86e-56011807f627","Type":"ContainerDied","Data":"541cdf339c2047ce3e4e032f6c2a6909917f1082b39aa9244c155736d6e6e937"}
Oct 30 19:29:44.604814 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.604720    5972 scope.go:122] "RemoveContainer" containerID="c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44"
Oct 30 19:29:44.616387 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.616365    5972 scope.go:122] "RemoveContainer" containerID="3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105"
Oct 30 19:29:44.623556 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.623539    5972 scope.go:122] "RemoveContainer" containerID="0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d"
Oct 30 19:29:44.630636 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.630623    5972 scope.go:122] "RemoveContainer" containerID="c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44"
Oct 30 19:29:44.630915 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:44.630887    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44\": container with ID starting with c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44 not found: ID does not exist" containerID="c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44"
Oct 30 19:29:44.630990 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.630923    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44"} err="failed to get container status \"c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44\": rpc error: code = NotFound desc = could not find container \"c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44\": container with ID starting with c42ad8e35508b33e9b4ea3626ee8933a6be71d0b9c840cad92283c0c933cda44 not found: ID does not exist"
Oct 30 19:29:44.630990 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.630947    5972 scope.go:122] "RemoveContainer" containerID="3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105"
Oct 30 19:29:44.631204 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:44.631185    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105\": container with ID starting with 3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105 not found: ID does not exist" containerID="3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105"
Oct 30 19:29:44.631268 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.631213    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105"} err="failed to get container status \"3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105\": rpc error: code = NotFound desc = could not find container \"3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105\": container with ID starting with 3d91cfa43cb359e09fa7db10654d524b28da172d7e01321ca8659ff5f097a105 not found: ID does not exist"
Oct 30 19:29:44.631268 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.631235    5972 scope.go:122] "RemoveContainer" containerID="0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d"
Oct 30 19:29:44.631513 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:44.631495    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d\": container with ID starting with 0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d not found: ID does not exist" containerID="0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d"
Oct 30 19:29:44.631564 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.631522    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d"} err="failed to get container status \"0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d\": rpc error: code = NotFound desc = could not find container \"0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d\": container with ID starting with 0c17ff41eb8eea97994e846229afb67bec89093ef24522865a3839a75c393d2d not found: ID does not exist"
Oct 30 19:29:44.685433 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.685406    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ae90e5-f14a-4bde-a86e-56011807f627-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:44.685517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.685435    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09ae90e5-f14a-4bde-a86e-56011807f627-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:44.685517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.685450    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:44.685517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.685461    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxk9v\" (UniqueName: \"kubernetes.io/projected/09ae90e5-f14a-4bde-a86e-56011807f627-kube-api-access-qxk9v\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:44.685517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.685475    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09ae90e5-f14a-4bde-a86e-56011807f627-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:44.795157 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.795076    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"]
Oct 30 19:29:44.799891 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:44.799867    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-pass-predictor-7d7b65d7f8-bphth"]
Oct 30 19:29:46.338303 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:29:46.338271    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" path="/var/lib/kubelet/pods/09ae90e5-f14a-4bde-a86e-56011807f627/volumes"
Oct 30 19:29:52.869429 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:29:52.869398    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:30:22.869830 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:30:22.869757    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:30:52.868687 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:30:52.868647    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:31:19.807976 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.807944    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"]
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808253    5972 cpu_manager.go:414] "containerMap: removing container" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kube-rbac-proxy"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808264    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kube-rbac-proxy"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808279    5972 cpu_manager.go:414] "containerMap: removing container" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="storage-initializer"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808286    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="storage-initializer"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808296    5972 cpu_manager.go:414] "containerMap: removing container" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kube-rbac-proxy"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808302    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kube-rbac-proxy"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808317    5972 cpu_manager.go:414] "containerMap: removing container" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808322    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808330    5972 cpu_manager.go:414] "containerMap: removing container" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808335    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808345    5972 cpu_manager.go:414] "containerMap: removing container" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="storage-initializer"
Oct 30 19:31:19.808404 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808351    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="storage-initializer"
Oct 30 19:31:19.808838 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808412    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kube-rbac-proxy"
Oct 30 19:31:19.808838 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808421    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kube-rbac-proxy"
Oct 30 19:31:19.808838 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808427    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="762f8357-e91d-4e91-80c7-b33637cc3347" containerName="kserve-container"
Oct 30 19:31:19.808838 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.808433    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ae90e5-f14a-4bde-a86e-56011807f627" containerName="kserve-container"
Oct 30 19:31:19.811484 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.811466    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.837707 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.837683    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.837830 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.837723    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.837830 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.837760    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.837905 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.837864    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbh5\" (UniqueName: \"kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.909844 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.909814    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:31:19.909989 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.909865    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Oct 30 19:31:19.909989 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.909928    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:31:19.909989 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.909935    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:31:19.909989 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.909956    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:31:19.909989 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.909957    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Oct 30 19:31:19.910920 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.910904    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"]
Oct 30 19:31:19.938609 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.938566    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbh5\" (UniqueName: \"kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.938748 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.938676    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.938748 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.938710    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.938864 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.938746    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.939143 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.939123    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.939340 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.939318    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.941517 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.941493    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:19.998381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:19.998356    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbh5\" (UniqueName: \"kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:20.128010 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:20.127940    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:20.542794 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:20.542757    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"]
Oct 30 19:31:20.545721 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:31:20.545692    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b59786f_4870_4ecd_89eb_83e1c6c8ac7d.slice/crio-f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8 WatchSource:0}: Error finding container f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8: Status 404 returned error can't find the container with id f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8
Oct 30 19:31:20.974187 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:20.974157    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" event={"ID":"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d","Type":"ContainerStarted","Data":"992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4"}
Oct 30 19:31:20.974662 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:20.974222    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" event={"ID":"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d","Type":"ContainerStarted","Data":"f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8"}
Oct 30 19:31:22.870894 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:31:22.870859    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:31:24.992110 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:24.992078    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/0.log"
Oct 30 19:31:24.992539 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:24.992115    5972 generic.go:356] "Generic (PLEG): container finished" podID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerID="992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4" exitCode=1
Oct 30 19:31:24.992539 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:24.992143    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" event={"ID":"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d","Type":"ContainerDied","Data":"992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4"}
Oct 30 19:31:25.998523 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:25.998494    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/0.log"
Oct 30 19:31:25.998987 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:25.998567    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" event={"ID":"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d","Type":"ContainerStarted","Data":"023487ba39c11a8d2c566411c144b0124de0100d427b10631b4285426decebae"}
Oct 30 19:31:28.009700 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.009669    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/1.log"
Oct 30 19:31:28.010170 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.010051    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/0.log"
Oct 30 19:31:28.010170 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.010085    5972 generic.go:356] "Generic (PLEG): container finished" podID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerID="023487ba39c11a8d2c566411c144b0124de0100d427b10631b4285426decebae" exitCode=1
Oct 30 19:31:28.010277 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.010166    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" event={"ID":"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d","Type":"ContainerDied","Data":"023487ba39c11a8d2c566411c144b0124de0100d427b10631b4285426decebae"}
Oct 30 19:31:28.010277 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.010238    5972 scope.go:122] "RemoveContainer" containerID="992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4"
Oct 30 19:31:28.010641 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.010618    5972 scope.go:122] "RemoveContainer" containerID="992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4"
Oct 30 19:31:28.021968 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:31:28.021939    5972 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_e2e-gw1-4a3552942a(4b59786f-4870-4ecd-89eb-83e1c6c8ac7d)\"" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d"
Oct 30 19:31:28.022087 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:31:28.021935    5972 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_e2e-gw1-4a3552942a_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d_0 in pod sandbox f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8 from index: no such id: '992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4'" containerID="992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4"
Oct 30 19:31:28.022087 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:28.021995    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_e2e-gw1-4a3552942a_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d_0 in pod sandbox f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8 from index: no such id: '992db5cc3700fb9d3f3e61417d076945621764bd92edc23675cab9e8ea639cf4'"
Oct 30 19:31:29.016225 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.016198    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/1.log"
Oct 30 19:31:29.467466 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.467420    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"]
Oct 30 19:31:29.605921 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.605899    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/1.log"
Oct 30 19:31:29.609450 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.609426    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:29.726649 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.726538    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location\") pod \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") "
Oct 30 19:31:29.726649 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.726632    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5\" (UniqueName: \"kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5\") pod \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") "
Oct 30 19:31:29.726887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.726701    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls\") pod \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") "
Oct 30 19:31:29.726887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.726735    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config\") pod \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\" (UID: \"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d\") "
Oct 30 19:31:29.726887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.726875    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location" pod "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" (UID: "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:29.727049 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.726979    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:29.727184 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.727165    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config" pod "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" (UID: "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:29.729071 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.729042    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls" pod "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" (UID: "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:29.729071 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.729060    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5" pod "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" (UID: "4b59786f-4870-4ecd-89eb-83e1c6c8ac7d"). InnerVolumeSpecName "kube-api-access-4cbh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:29.828020 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.827974    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:29.828020 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.828014    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:29.828020 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:29.828027    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbh5\" (UniqueName: \"kubernetes.io/projected/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d-kube-api-access-4cbh5\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:30.029467 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.029382    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr_4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/storage-initializer/1.log"
Oct 30 19:31:30.029887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.029490    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr" event={"ID":"4b59786f-4870-4ecd-89eb-83e1c6c8ac7d","Type":"ContainerDied","Data":"f1acbd5be6b2c6512378154a4f4869a501af0a72b635eb1a5662ee772b1818e8"}
Oct 30 19:31:30.029887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.029519    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"
Oct 30 19:31:30.029887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.029658    5972 scope.go:122] "RemoveContainer" containerID="023487ba39c11a8d2c566411c144b0124de0100d427b10631b4285426decebae"
Oct 30 19:31:30.254286 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.254251    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"]
Oct 30 19:31:30.260377 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.260341    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-fail-predictor-94f56d78-kg9jr"]
Oct 30 19:31:30.340063 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:31:30.339983    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" path="/var/lib/kubelet/pods/4b59786f-4870-4ecd-89eb-83e1c6c8ac7d/volumes"
Oct 30 19:31:52.872933 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:31:52.872845    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:32:22.869294 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:32:22.869257    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:32:52.869931 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:32:52.869898    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:33:05.284534 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284501    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"]
Oct 30 19:33:05.284973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284829    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerName="storage-initializer"
Oct 30 19:33:05.284973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284840    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerName="storage-initializer"
Oct 30 19:33:05.284973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284850    5972 cpu_manager.go:414] "containerMap: removing container" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerName="storage-initializer"
Oct 30 19:33:05.284973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284855    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerName="storage-initializer"
Oct 30 19:33:05.284973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284923    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerName="storage-initializer"
Oct 30 19:33:05.284973 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.284930    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b59786f-4870-4ecd-89eb-83e1c6c8ac7d" containerName="storage-initializer"
Oct 30 19:33:05.288155 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.288134    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.338937 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.338906    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Oct 30 19:33:05.339042 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.338945    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:33:05.358450 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.358427    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.358518 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.358490    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrtn\" (UniqueName: \"kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.358569 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.358551    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.358644 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.358608    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.386420 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.386392    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:33:05.386420 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.386392    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Oct 30 19:33:05.386614 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.386432    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:33:05.386614 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.386432    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:33:05.387469 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.387447    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"]
Oct 30 19:33:05.459852 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.459821    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.459985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.459886    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrtn\" (UniqueName: \"kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.459985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.459918    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.459985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.459957    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.460259 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.460234    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.460540 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.460521    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.462436 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.462419    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.519244 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.519214    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrtn\" (UniqueName: \"kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:05.604431 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:05.604347    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:06.022116 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:06.022084    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"]
Oct 30 19:33:06.025413 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:33:06.025381    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f7fca7_901a_4c37_a5a9_0fc06c60c127.slice/crio-ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c WatchSource:0}: Error finding container ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c: Status 404 returned error can't find the container with id ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c
Oct 30 19:33:06.406381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:06.406292    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" event={"ID":"a6f7fca7-901a-4c37-a5a9-0fc06c60c127","Type":"ContainerStarted","Data":"b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97"}
Oct 30 19:33:06.406381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:06.406375    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" event={"ID":"a6f7fca7-901a-4c37-a5a9-0fc06c60c127","Type":"ContainerStarted","Data":"ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c"}
Oct 30 19:33:09.419290 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:09.419268    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/0.log"
Oct 30 19:33:09.419740 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:09.419307    5972 generic.go:356] "Generic (PLEG): container finished" podID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerID="b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97" exitCode=1
Oct 30 19:33:09.419740 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:09.419365    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" event={"ID":"a6f7fca7-901a-4c37-a5a9-0fc06c60c127","Type":"ContainerDied","Data":"b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97"}
Oct 30 19:33:10.425187 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:10.425163    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/0.log"
Oct 30 19:33:10.425555 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:10.425234    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" event={"ID":"a6f7fca7-901a-4c37-a5a9-0fc06c60c127","Type":"ContainerStarted","Data":"d8cd6ad986b7a0163628e3ebac2221201bd973174c98e7ec00f70548e5c1c349"}
Oct 30 19:33:14.442789 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.442764    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/1.log"
Oct 30 19:33:14.443202 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.443188    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/0.log"
Oct 30 19:33:14.443250 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.443222    5972 generic.go:356] "Generic (PLEG): container finished" podID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerID="d8cd6ad986b7a0163628e3ebac2221201bd973174c98e7ec00f70548e5c1c349" exitCode=1
Oct 30 19:33:14.443288 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.443272    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" event={"ID":"a6f7fca7-901a-4c37-a5a9-0fc06c60c127","Type":"ContainerDied","Data":"d8cd6ad986b7a0163628e3ebac2221201bd973174c98e7ec00f70548e5c1c349"}
Oct 30 19:33:14.443336 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.443322    5972 scope.go:122] "RemoveContainer" containerID="b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97"
Oct 30 19:33:14.443683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.443658    5972 scope.go:122] "RemoveContainer" containerID="b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97"
Oct 30 19:33:14.453826 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:33:14.453806    5972 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_e2e-gw1-4a3552942a(a6f7fca7-901a-4c37-a5a9-0fc06c60c127)\"" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127"
Oct 30 19:33:14.453993 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:33:14.453970    5972 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_e2e-gw1-4a3552942a_a6f7fca7-901a-4c37-a5a9-0fc06c60c127_0 in pod sandbox ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c from index: no such id: 'b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97'" containerID="b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97"
Oct 30 19:33:14.454063 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:14.454003    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_e2e-gw1-4a3552942a_a6f7fca7-901a-4c37-a5a9-0fc06c60c127_0 in pod sandbox ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c from index: no such id: 'b9b1d18c462caf360abed94eeb59ba4ee747e869c463c9c8549ca6f9ef87ce97'"
Oct 30 19:33:15.448260 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:15.448237    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/1.log"
Oct 30 19:33:22.869224 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:33:22.869137    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:33:25.261058 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.261024    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"]
Oct 30 19:33:25.382705 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.382675    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/1.log"
Oct 30 19:33:25.386250 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.386226    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:25.487283 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.487260    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9_a6f7fca7-901a-4c37-a5a9-0fc06c60c127/storage-initializer/1.log"
Oct 30 19:33:25.487403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.487356    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"
Oct 30 19:33:25.487403 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.487357    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9" event={"ID":"a6f7fca7-901a-4c37-a5a9-0fc06c60c127","Type":"ContainerDied","Data":"ed46e3f7158c0e67e9c52b56b84f9d69dfa94ae9520eff4834d1d12c83293c3c"}
Oct 30 19:33:25.487481 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.487452    5972 scope.go:122] "RemoveContainer" containerID="d8cd6ad986b7a0163628e3ebac2221201bd973174c98e7ec00f70548e5c1c349"
Oct 30 19:33:25.527808 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.527754    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn\" (UniqueName: \"kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn\") pod \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") "
Oct 30 19:33:25.527910 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.527813    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls\") pod \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") "
Oct 30 19:33:25.527910 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.527852    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config\") pod \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") "
Oct 30 19:33:25.528026 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.528002    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location\") pod \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\" (UID: \"a6f7fca7-901a-4c37-a5a9-0fc06c60c127\") "
Oct 30 19:33:25.528267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.528237    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location" pod "a6f7fca7-901a-4c37-a5a9-0fc06c60c127" (UID: "a6f7fca7-901a-4c37-a5a9-0fc06c60c127"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:25.528267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.528248    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config" pod "a6f7fca7-901a-4c37-a5a9-0fc06c60c127" (UID: "a6f7fca7-901a-4c37-a5a9-0fc06c60c127"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:25.529887 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.529866    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls" pod "a6f7fca7-901a-4c37-a5a9-0fc06c60c127" (UID: "a6f7fca7-901a-4c37-a5a9-0fc06c60c127"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:25.529983 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.529963    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn" pod "a6f7fca7-901a-4c37-a5a9-0fc06c60c127" (UID: "a6f7fca7-901a-4c37-a5a9-0fc06c60c127"). InnerVolumeSpecName "kube-api-access-mjrtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:25.629243 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.629202    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.629243 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.629233    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrtn\" (UniqueName: \"kubernetes.io/projected/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-api-access-mjrtn\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.629243 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.629244    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.629488 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.629253    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f7fca7-901a-4c37-a5a9-0fc06c60c127-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:25.967044 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.967012    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"]
Oct 30 19:33:25.974026 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:25.973994    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-fail-predictor-7d8588d4df-wwcz9"]
Oct 30 19:33:26.338129 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:26.338101    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" path="/var/lib/kubelet/pods/a6f7fca7-901a-4c37-a5a9-0fc06c60c127/volumes"
Oct 30 19:33:29.892572 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892540    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"]
Oct 30 19:33:29.893034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892868    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerName="storage-initializer"
Oct 30 19:33:29.893034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892879    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerName="storage-initializer"
Oct 30 19:33:29.893034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892893    5972 cpu_manager.go:414] "containerMap: removing container" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerName="storage-initializer"
Oct 30 19:33:29.893034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892898    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerName="storage-initializer"
Oct 30 19:33:29.893034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892954    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerName="storage-initializer"
Oct 30 19:33:29.893034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.892964    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f7fca7-901a-4c37-a5a9-0fc06c60c127" containerName="storage-initializer"
Oct 30 19:33:29.897444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.897425    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:29.996433 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996403    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:33:29.996433 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996420    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:33:29.996433 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996422    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Oct 30 19:33:29.996718 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996455    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-paddle-predictor-serving-cert\""
Oct 30 19:33:29.996718 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996415    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:33:29.996718 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996458    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:33:29.996821 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.996738    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:33:29.997860 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:29.997843    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"]
Oct 30 19:33:30.067089 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.067062    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.067226 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.067126    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.067278 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.067234    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.067278 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.067258    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmb4h\" (UniqueName: \"kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.067359 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.067299    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.168429 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.168359    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.168542 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.168451    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.168542 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.168482    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.168542 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.168507    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmb4h\" (UniqueName: \"kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.168782 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.168682    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.169100 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.169075    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.169225 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.169120    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.169225 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.169209    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.171033 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.171018    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.228223 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.228192    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmb4h\" (UniqueName: \"kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h\") pod \"isvc-paddle-predictor-6f6c56b897-7vmdn\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") " pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.514169 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.514140    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:30.933281 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:30.933211    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"]
Oct 30 19:33:30.936825 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:33:30.936800    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5056095_7435_42ba_a62e_aa447537c527.slice/crio-97db7681f7763c221a40e5f1992d219c15aa3da6ad2a82256cf207dcd9a0ae17 WatchSource:0}: Error finding container 97db7681f7763c221a40e5f1992d219c15aa3da6ad2a82256cf207dcd9a0ae17: Status 404 returned error can't find the container with id 97db7681f7763c221a40e5f1992d219c15aa3da6ad2a82256cf207dcd9a0ae17
Oct 30 19:33:31.512440 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:31.512404    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerStarted","Data":"c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea"}
Oct 30 19:33:31.512632 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:31.512463    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerStarted","Data":"97db7681f7763c221a40e5f1992d219c15aa3da6ad2a82256cf207dcd9a0ae17"}
Oct 30 19:33:36.534192 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:36.534157    5972 generic.go:356] "Generic (PLEG): container finished" podID="b5056095-7435-42ba-a62e-aa447537c527" containerID="c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea" exitCode=0
Oct 30 19:33:36.534611 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:36.534234    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerDied","Data":"c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea"}
Oct 30 19:33:36.535502 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:36.535489    5972 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:33:49.594332 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:49.594301    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerStarted","Data":"29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289"}
Oct 30 19:33:49.594716 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:49.594362    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerStarted","Data":"827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6"}
Oct 30 19:33:49.594716 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:49.594641    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:49.594787 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:49.594775    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:49.596013 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:49.595989    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 19:33:49.763257 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:49.763208    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podStartSLOduration=3.545497529 podStartE2EDuration="20.763195721s" totalImagesPullingTime="12.478315787s" totalInitContainerRuntime="4.739382405s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:49.762763918 +0000 UTC m=+3279.554731205" watchObservedRunningTime="2026-10-30 19:33:49.763195721 +0000 UTC m=+3279.555162989"
Oct 30 19:33:50.598709 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:50.598668    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 19:33:52.869777 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:33:52.869745    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:33:55.603796 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:55.603771    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:33:55.604252 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:33:55.604227    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 19:34:05.605006 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:05.604967    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 19:34:10.687804 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:10.687776    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:34:10.706246 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:10.706220    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:34:15.605079 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:15.605035    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 19:34:22.869364 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:34:22.869325    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:34:25.604815 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:25.604781    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 19:34:35.605245 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:35.605214    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:34:41.191771 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:41.191741    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"]
Oct 30 19:34:41.192221 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:41.191962    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container" containerID="cri-o://827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6" gracePeriod=30
Oct 30 19:34:41.192221 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:41.192008    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kube-rbac-proxy" containerID="cri-o://29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289" gracePeriod=30
Oct 30 19:34:41.806106 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:41.806076    5972 generic.go:356] "Generic (PLEG): container finished" podID="b5056095-7435-42ba-a62e-aa447537c527" containerID="29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289" exitCode=0
Oct 30 19:34:41.806307 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:41.806157    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerDied","Data":"29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289"}
Oct 30 19:34:44.032557 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.032534    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:34:44.107042 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.106969    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location\") pod \"b5056095-7435-42ba-a62e-aa447537c527\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") "
Oct 30 19:34:44.107042 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.107025    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h\" (UniqueName: \"kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h\") pod \"b5056095-7435-42ba-a62e-aa447537c527\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") "
Oct 30 19:34:44.107260 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.107067    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config\") pod \"b5056095-7435-42ba-a62e-aa447537c527\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") "
Oct 30 19:34:44.107260 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.107091    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert\") pod \"b5056095-7435-42ba-a62e-aa447537c527\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") "
Oct 30 19:34:44.107260 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.107165    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls\") pod \"b5056095-7435-42ba-a62e-aa447537c527\" (UID: \"b5056095-7435-42ba-a62e-aa447537c527\") "
Oct 30 19:34:44.107498 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.107437    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config" pod "b5056095-7435-42ba-a62e-aa447537c527" (UID: "b5056095-7435-42ba-a62e-aa447537c527"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:44.107498 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.107477    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert" pod "b5056095-7435-42ba-a62e-aa447537c527" (UID: "b5056095-7435-42ba-a62e-aa447537c527"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:44.109222 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.109200    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls" pod "b5056095-7435-42ba-a62e-aa447537c527" (UID: "b5056095-7435-42ba-a62e-aa447537c527"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:44.109299 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.109228    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h" pod "b5056095-7435-42ba-a62e-aa447537c527" (UID: "b5056095-7435-42ba-a62e-aa447537c527"). InnerVolumeSpecName "kube-api-access-jmb4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:44.121278 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.121251    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location" pod "b5056095-7435-42ba-a62e-aa447537c527" (UID: "b5056095-7435-42ba-a62e-aa447537c527"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:44.208290 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.208269    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5056095-7435-42ba-a62e-aa447537c527-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.208290 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.208290    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmb4h\" (UniqueName: \"kubernetes.io/projected/b5056095-7435-42ba-a62e-aa447537c527-kube-api-access-jmb4h\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.208437 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.208300    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.208437 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.208309    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5056095-7435-42ba-a62e-aa447537c527-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.208437 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.208318    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5056095-7435-42ba-a62e-aa447537c527-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.821082 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.821050    5972 generic.go:356] "Generic (PLEG): container finished" podID="b5056095-7435-42ba-a62e-aa447537c527" containerID="827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6" exitCode=0
Oct 30 19:34:44.821284 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.821104    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"
Oct 30 19:34:44.821284 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.821113    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerDied","Data":"827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6"}
Oct 30 19:34:44.821284 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.821175    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn" event={"ID":"b5056095-7435-42ba-a62e-aa447537c527","Type":"ContainerDied","Data":"97db7681f7763c221a40e5f1992d219c15aa3da6ad2a82256cf207dcd9a0ae17"}
Oct 30 19:34:44.821284 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.821212    5972 scope.go:122] "RemoveContainer" containerID="29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289"
Oct 30 19:34:44.829385 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.829355    5972 scope.go:122] "RemoveContainer" containerID="827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6"
Oct 30 19:34:44.836764 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.836748    5972 scope.go:122] "RemoveContainer" containerID="c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea"
Oct 30 19:34:44.843789 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.843773    5972 scope.go:122] "RemoveContainer" containerID="29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289"
Oct 30 19:34:44.844027 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:34:44.844011    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289\": container with ID starting with 29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289 not found: ID does not exist" containerID="29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289"
Oct 30 19:34:44.844078 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.844035    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289"} err="failed to get container status \"29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289\": rpc error: code = NotFound desc = could not find container \"29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289\": container with ID starting with 29d4a8c4fbbb31c68e30bd8be64082d1e99a3e89ec3c30feb85397f4a8ea8289 not found: ID does not exist"
Oct 30 19:34:44.844078 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.844052    5972 scope.go:122] "RemoveContainer" containerID="827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6"
Oct 30 19:34:44.844278 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:34:44.844259    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6\": container with ID starting with 827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6 not found: ID does not exist" containerID="827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6"
Oct 30 19:34:44.844316 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.844285    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6"} err="failed to get container status \"827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6\": rpc error: code = NotFound desc = could not find container \"827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6\": container with ID starting with 827d9f8c749b31ff5845431c1a4eed9a08980b41a2f6d51943fde2ede4bab1c6 not found: ID does not exist"
Oct 30 19:34:44.844316 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.844299    5972 scope.go:122] "RemoveContainer" containerID="c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea"
Oct 30 19:34:44.844498 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:34:44.844484    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea\": container with ID starting with c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea not found: ID does not exist" containerID="c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea"
Oct 30 19:34:44.844536 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.844501    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea"} err="failed to get container status \"c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea\": rpc error: code = NotFound desc = could not find container \"c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea\": container with ID starting with c059db09b79d1c151ace7dbeb1ce660ea3c8ff95e6c3ba333fb4cd570eb0b6ea not found: ID does not exist"
Oct 30 19:34:44.980783 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:44.980753    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"]
Oct 30 19:34:45.028691 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:45.028668    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-predictor-6f6c56b897-7vmdn"]
Oct 30 19:34:46.338570 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:34:46.338539    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5056095-7435-42ba-a62e-aa447537c527" path="/var/lib/kubelet/pods/b5056095-7435-42ba-a62e-aa447537c527/volumes"
Oct 30 19:34:52.871152 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:34:52.871066    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:35:22.870325 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:35:22.870287    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:35:52.870209 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:35:52.870168    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:36:06.136444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136406    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"]
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136893    5972 cpu_manager.go:414] "containerMap: removing container" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kube-rbac-proxy"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136911    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kube-rbac-proxy"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136939    5972 cpu_manager.go:414] "containerMap: removing container" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="storage-initializer"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136946    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="storage-initializer"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136979    5972 cpu_manager.go:414] "containerMap: removing container" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.136987    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.137071    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kserve-container"
Oct 30 19:36:06.137085 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.137084    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5056095-7435-42ba-a62e-aa447537c527" containerName="kube-rbac-proxy"
Oct 30 19:36:06.140409 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.140389    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.191654 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.191626    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:36:06.191654 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.191648    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:36:06.239660 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.239626    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:36:06.239845 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.239686    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Oct 30 19:36:06.239845 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.239690    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:36:06.239845 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.239690    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 19:36:06.239845 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.239800    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:36:06.240891 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.240875    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"]
Oct 30 19:36:06.280180 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.280146    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.280381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.280190    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.280381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.280303    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89t4x\" (UniqueName: \"kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.280381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.280364    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.280504 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.280425    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.380874 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.380838    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.380934    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.380957    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89t4x\" (UniqueName: \"kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.380985    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381052 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.381002    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381323 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.381298    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381572 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.381545    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.381698 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.381676    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.383608 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.383575    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.440745 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.440727    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89t4x\" (UniqueName: \"kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x\") pod \"isvc-paddle-v2-kserve-predictor-98987845-92spc\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") " pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.456653 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.456632    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:06.869141 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:06.869064    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"]
Oct 30 19:36:06.872549 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:36:06.872516    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245e7efb_51a1_4384_aaad_4aedeabac2d8.slice/crio-b223336f58124be78e3e460311ee66947141ed8e2f0f5d412cb8c2cd5bea94ad WatchSource:0}: Error finding container b223336f58124be78e3e460311ee66947141ed8e2f0f5d412cb8c2cd5bea94ad: Status 404 returned error can't find the container with id b223336f58124be78e3e460311ee66947141ed8e2f0f5d412cb8c2cd5bea94ad
Oct 30 19:36:07.135777 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:07.135693    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerStarted","Data":"a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d"}
Oct 30 19:36:07.135777 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:07.135748    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerStarted","Data":"b223336f58124be78e3e460311ee66947141ed8e2f0f5d412cb8c2cd5bea94ad"}
Oct 30 19:36:12.155778 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:12.155745    5972 generic.go:356] "Generic (PLEG): container finished" podID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerID="a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d" exitCode=0
Oct 30 19:36:12.156188 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:12.155804    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerDied","Data":"a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d"}
Oct 30 19:36:13.161533 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:13.161502    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerStarted","Data":"c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197"}
Oct 30 19:36:13.161974 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:13.161558    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerStarted","Data":"716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7"}
Oct 30 19:36:13.161974 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:13.161837    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:13.161974 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:13.161970    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:13.163206 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:13.163183    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:36:13.279356 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:13.279307    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podStartSLOduration=2.150175814 podStartE2EDuration="7.279294612s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.129118798s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:13.277813343 +0000 UTC m=+3423.069780632" watchObservedRunningTime="2026-10-30 19:36:13.279294612 +0000 UTC m=+3423.071261900"
Oct 30 19:36:14.166166 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:14.166127    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:36:19.171472 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:19.171391    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:36:19.172050 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:19.172022    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:36:22.869290 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:36:22.869249    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:36:29.172025 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:29.171990    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:36:39.172104 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:39.172065    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:36:49.172683 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:49.172638    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:36:52.869532 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:36:52.869494    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:36:59.172754 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:36:59.172724    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:37:08.684852 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:08.684824    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"]
Oct 30 19:37:08.685363 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:08.685033    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" containerID="cri-o://716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7" gracePeriod=30
Oct 30 19:37:08.685363 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:08.685077    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kube-rbac-proxy" containerID="cri-o://c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197" gracePeriod=30
Oct 30 19:37:09.167172 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:09.167132    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Oct 30 19:37:09.172557 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:09.172536    5972 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 19:37:09.381995 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:09.381962    5972 generic.go:356] "Generic (PLEG): container finished" podID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerID="c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197" exitCode=0
Oct 30 19:37:09.382146 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:09.382025    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerDied","Data":"c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197"}
Oct 30 19:37:11.523316 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.523288    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:37:11.636028 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.635959    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location\") pod \"245e7efb-51a1-4384-aaad-4aedeabac2d8\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") "
Oct 30 19:37:11.636028 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.635998    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config\") pod \"245e7efb-51a1-4384-aaad-4aedeabac2d8\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") "
Oct 30 19:37:11.636201 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.636033    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x\" (UniqueName: \"kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x\") pod \"245e7efb-51a1-4384-aaad-4aedeabac2d8\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") "
Oct 30 19:37:11.636201 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.636058    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls\") pod \"245e7efb-51a1-4384-aaad-4aedeabac2d8\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") "
Oct 30 19:37:11.636308 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.636220    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert\") pod \"245e7efb-51a1-4384-aaad-4aedeabac2d8\" (UID: \"245e7efb-51a1-4384-aaad-4aedeabac2d8\") "
Oct 30 19:37:11.636400 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.636380    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config" pod "245e7efb-51a1-4384-aaad-4aedeabac2d8" (UID: "245e7efb-51a1-4384-aaad-4aedeabac2d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:11.636572 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.636547    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert" pod "245e7efb-51a1-4384-aaad-4aedeabac2d8" (UID: "245e7efb-51a1-4384-aaad-4aedeabac2d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:11.636671 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.636604    5972 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:11.638298 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.638275    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls" pod "245e7efb-51a1-4384-aaad-4aedeabac2d8" (UID: "245e7efb-51a1-4384-aaad-4aedeabac2d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:11.638381 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.638360    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x" pod "245e7efb-51a1-4384-aaad-4aedeabac2d8" (UID: "245e7efb-51a1-4384-aaad-4aedeabac2d8"). InnerVolumeSpecName "kube-api-access-89t4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:11.645812 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.645789    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location" pod "245e7efb-51a1-4384-aaad-4aedeabac2d8" (UID: "245e7efb-51a1-4384-aaad-4aedeabac2d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:11.737444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.737418    5972 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245e7efb-51a1-4384-aaad-4aedeabac2d8-cabundle-cert\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:11.737444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.737441    5972 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245e7efb-51a1-4384-aaad-4aedeabac2d8-kserve-provision-location\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:11.737444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.737451    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89t4x\" (UniqueName: \"kubernetes.io/projected/245e7efb-51a1-4384-aaad-4aedeabac2d8-kube-api-access-89t4x\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:11.737675 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:11.737460    5972 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245e7efb-51a1-4384-aaad-4aedeabac2d8-proxy-tls\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:12.395121 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.395087    5972 generic.go:356] "Generic (PLEG): container finished" podID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerID="716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7" exitCode=0
Oct 30 19:37:12.395267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.395159    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerDied","Data":"716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7"}
Oct 30 19:37:12.395267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.395165    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"
Oct 30 19:37:12.395267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.395225    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc" event={"ID":"245e7efb-51a1-4384-aaad-4aedeabac2d8","Type":"ContainerDied","Data":"b223336f58124be78e3e460311ee66947141ed8e2f0f5d412cb8c2cd5bea94ad"}
Oct 30 19:37:12.395267 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.395262    5972 scope.go:122] "RemoveContainer" containerID="c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197"
Oct 30 19:37:12.403770 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.403567    5972 scope.go:122] "RemoveContainer" containerID="716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7"
Oct 30 19:37:12.410929 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.410906    5972 scope.go:122] "RemoveContainer" containerID="a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d"
Oct 30 19:37:12.418136 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.418121    5972 scope.go:122] "RemoveContainer" containerID="c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197"
Oct 30 19:37:12.418412 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:37:12.418382    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197\": container with ID starting with c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197 not found: ID does not exist" containerID="c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197"
Oct 30 19:37:12.418489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.418414    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197"} err="failed to get container status \"c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197\": rpc error: code = NotFound desc = could not find container \"c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197\": container with ID starting with c4e1c0b699d2b76698817e54d3e5bd8414e99665ca5d0acbd2f8b9f37e199197 not found: ID does not exist"
Oct 30 19:37:12.418489 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.418438    5972 scope.go:122] "RemoveContainer" containerID="716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7"
Oct 30 19:37:12.418691 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:37:12.418673    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7\": container with ID starting with 716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7 not found: ID does not exist" containerID="716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7"
Oct 30 19:37:12.418739 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.418698    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7"} err="failed to get container status \"716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7\": rpc error: code = NotFound desc = could not find container \"716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7\": container with ID starting with 716e88f214b533255aee12e3d4f6aacd761e6f3567e78973dbf394b15d33e7d7 not found: ID does not exist"
Oct 30 19:37:12.418739 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.418714    5972 scope.go:122] "RemoveContainer" containerID="a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d"
Oct 30 19:37:12.418947 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:37:12.418930    5972 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d\": container with ID starting with a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d not found: ID does not exist" containerID="a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d"
Oct 30 19:37:12.419004 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.418957    5972 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d"} err="failed to get container status \"a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d\": rpc error: code = NotFound desc = could not find container \"a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d\": container with ID starting with a0b23d8f3f5b3af87377c949ddb8d7c8e8f2a4c7da6053a0271094334121865d not found: ID does not exist"
Oct 30 19:37:12.602417 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.602350    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"]
Oct 30 19:37:12.650725 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:12.650698    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-v2-kserve-predictor-98987845-92spc"]
Oct 30 19:37:14.339086 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:37:14.339051    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" path="/var/lib/kubelet/pods/245e7efb-51a1-4384-aaad-4aedeabac2d8/volumes"
Oct 30 19:37:22.869108 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:37:22.869076    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:37:52.870972 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:37:52.870876    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:38:22.869862 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:38:22.869820    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:38:52.870971 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:38:52.870926    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:39:10.718548 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:10.718524    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:39:10.734350 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:10.734331    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shstn_0808db1b-d15b-47c9-aa62-8a1a6309398b/ovn-acl-logging/0.log"
Oct 30 19:39:22.869660 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:39:22.869556    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:39:39.420120 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420087    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zbvd2/must-gather-9j7vw"]
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420389    5972 cpu_manager.go:414] "containerMap: removing container" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420399    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420413    5972 cpu_manager.go:414] "containerMap: removing container" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kube-rbac-proxy"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420418    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kube-rbac-proxy"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420435    5972 cpu_manager.go:414] "containerMap: removing container" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="storage-initializer"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420441    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="storage-initializer"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420510    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kube-rbac-proxy"
Oct 30 19:39:39.420664 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.420523    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="245e7efb-51a1-4384-aaad-4aedeabac2d8" containerName="kserve-container"
Oct 30 19:39:39.423538 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.423521    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.497974 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.497949    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output\") pod \"must-gather-9j7vw\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") " pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.498101 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.497984    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwk4d\" (UniqueName: \"kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d\") pod \"must-gather-9j7vw\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") " pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.524258 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.524238    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zbvd2\"/\"kube-root-ca.crt\""
Oct 30 19:39:39.524258 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.524242    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zbvd2\"/\"openshift-service-ca.crt\""
Oct 30 19:39:39.524421 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.524279    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zbvd2\"/\"default-dockercfg-4hgnw\""
Oct 30 19:39:39.525021 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.525003    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zbvd2/must-gather-9j7vw"]
Oct 30 19:39:39.599087 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.599062    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output\") pod \"must-gather-9j7vw\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") " pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.599201 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.599098    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwk4d\" (UniqueName: \"kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d\") pod \"must-gather-9j7vw\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") " pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.599420 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.599399    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output\") pod \"must-gather-9j7vw\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") " pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.658196 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.658177    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwk4d\" (UniqueName: \"kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d\") pod \"must-gather-9j7vw\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") " pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:39.744540 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:39.744514    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:39:40.119310 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:40.119221    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zbvd2/must-gather-9j7vw"]
Oct 30 19:39:40.126330 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:39:40.123442    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf728d41e_03d6_4181_9abb_f0a21ff2c26b.slice/crio-c36159b53a10e889c90934d59bb6e6398d9e8972b18b79118b06bbaeea55c6c4 WatchSource:0}: Error finding container c36159b53a10e889c90934d59bb6e6398d9e8972b18b79118b06bbaeea55c6c4: Status 404 returned error can't find the container with id c36159b53a10e889c90934d59bb6e6398d9e8972b18b79118b06bbaeea55c6c4
Oct 30 19:39:40.129241 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:40.129221    5972 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:39:40.973574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:40.973542    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zbvd2/must-gather-9j7vw" event={"ID":"f728d41e-03d6-4181-9abb-f0a21ff2c26b","Type":"ContainerStarted","Data":"c36159b53a10e889c90934d59bb6e6398d9e8972b18b79118b06bbaeea55c6c4"}
Oct 30 19:39:47.001618 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:47.001560    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zbvd2/must-gather-9j7vw" event={"ID":"f728d41e-03d6-4181-9abb-f0a21ff2c26b","Type":"ContainerStarted","Data":"412cbd9325c7d07165c2031d1b17b78e7b8f19890a61ec826ebb59f236f16431"}
Oct 30 19:39:48.007314 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:48.007274    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zbvd2/must-gather-9j7vw" event={"ID":"f728d41e-03d6-4181-9abb-f0a21ff2c26b","Type":"ContainerStarted","Data":"af8ac8b349c7300bb3c30719535f3b349a104f17afbf85cca3c97047cef476ec"}
Oct 30 19:39:48.119564 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:39:48.119510    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zbvd2/must-gather-9j7vw" podStartSLOduration=2.49378276 podStartE2EDuration="9.11949655s" totalImagesPullingTime="6.62571379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:48.118843333 +0000 UTC m=+3637.910810621" watchObservedRunningTime="2026-10-30 19:39:48.11949655 +0000 UTC m=+3637.911463829"
Oct 30 19:39:52.871208 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:39:52.871173    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:40:22.876605 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:40:22.876544    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:40:52.872491 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:40:52.872401    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:40:53.283465 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:40:53.283427    5972 generic.go:356] "Generic (PLEG): container finished" podID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerID="412cbd9325c7d07165c2031d1b17b78e7b8f19890a61ec826ebb59f236f16431" exitCode=0
Oct 30 19:40:53.283742 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:40:53.283476    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zbvd2/must-gather-9j7vw" event={"ID":"f728d41e-03d6-4181-9abb-f0a21ff2c26b","Type":"ContainerDied","Data":"412cbd9325c7d07165c2031d1b17b78e7b8f19890a61ec826ebb59f236f16431"}
Oct 30 19:40:53.283909 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:40:53.283891    5972 scope.go:122] "RemoveContainer" containerID="412cbd9325c7d07165c2031d1b17b78e7b8f19890a61ec826ebb59f236f16431"
Oct 30 19:40:53.742427 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:40:53.742395    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zbvd2_must-gather-9j7vw_f728d41e-03d6-4181-9abb-f0a21ff2c26b/gather/0.log"
Oct 30 19:41:00.169158 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.169116    5972 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zbvd2/must-gather-9j7vw"]
Oct 30 19:41:00.169730 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.169321    5972 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-zbvd2/must-gather-9j7vw" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="copy" containerID="cri-o://af8ac8b349c7300bb3c30719535f3b349a104f17afbf85cca3c97047cef476ec" gracePeriod=2
Oct 30 19:41:00.216889 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.216842    5972 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zbvd2/must-gather-9j7vw"]
Oct 30 19:41:00.218694 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:41:00.218646    5972 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-9j7vw\" is forbidden: User \"system:node:ip-10-0-134-133.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zbvd2\": no relationship found between node 'ip-10-0-134-133.us-west-2.compute.internal' and this object" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:41:00.316247 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.316219    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zbvd2_must-gather-9j7vw_f728d41e-03d6-4181-9abb-f0a21ff2c26b/copy/0.log"
Oct 30 19:41:00.316676 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.316652    5972 generic.go:356] "Generic (PLEG): container finished" podID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerID="af8ac8b349c7300bb3c30719535f3b349a104f17afbf85cca3c97047cef476ec" exitCode=143
Oct 30 19:41:00.385985 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:41:00.385943    5972 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-9j7vw\" is forbidden: User \"system:node:ip-10-0-134-133.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zbvd2\": no relationship found between node 'ip-10-0-134-133.us-west-2.compute.internal' and this object" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:41:00.458444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.458410    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zbvd2_must-gather-9j7vw_f728d41e-03d6-4181-9abb-f0a21ff2c26b/copy/0.log"
Oct 30 19:41:00.460299 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.460277    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:41:00.533123 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.533084    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output\") pod \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") "
Oct 30 19:41:00.533304 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.533165    5972 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d\" (UniqueName: \"kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d\") pod \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\" (UID: \"f728d41e-03d6-4181-9abb-f0a21ff2c26b\") "
Oct 30 19:41:00.534498 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.534465    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output" pod "f728d41e-03d6-4181-9abb-f0a21ff2c26b" (UID: "f728d41e-03d6-4181-9abb-f0a21ff2c26b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:00.535666 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.535642    5972 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d" pod "f728d41e-03d6-4181-9abb-f0a21ff2c26b" (UID: "f728d41e-03d6-4181-9abb-f0a21ff2c26b"). InnerVolumeSpecName "kube-api-access-dwk4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:00.633848 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.633810    5972 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwk4d\" (UniqueName: \"kubernetes.io/projected/f728d41e-03d6-4181-9abb-f0a21ff2c26b-kube-api-access-dwk4d\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.633848 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:00.633842    5972 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f728d41e-03d6-4181-9abb-f0a21ff2c26b-must-gather-output\") on node \"ip-10-0-134-133.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:01.322158 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:01.322131    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zbvd2_must-gather-9j7vw_f728d41e-03d6-4181-9abb-f0a21ff2c26b/copy/0.log"
Oct 30 19:41:01.322628 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:01.322562    5972 scope.go:122] "RemoveContainer" containerID="af8ac8b349c7300bb3c30719535f3b349a104f17afbf85cca3c97047cef476ec"
Oct 30 19:41:01.322628 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:01.322600    5972 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zbvd2/must-gather-9j7vw"
Oct 30 19:41:01.331390 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:01.331366    5972 scope.go:122] "RemoveContainer" containerID="412cbd9325c7d07165c2031d1b17b78e7b8f19890a61ec826ebb59f236f16431"
Oct 30 19:41:02.338360 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:02.338324    5972 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" path="/var/lib/kubelet/pods/f728d41e-03d6-4181-9abb-f0a21ff2c26b/volumes"
Oct 30 19:41:04.347528 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:04.347494    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-77qht_8ac3d6d9-3493-46b4-a5df-e1a2b3e7585b/global-pull-secret-syncer/0.log"
Oct 30 19:41:08.582190 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:08.582159    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fdf4b_5f1b72b8-3c48-4426-b83c-94bc84b69bf4/konnectivity-agent/0.log"
Oct 30 19:41:09.554752 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:09.554718    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-133.us-west-2.compute.internal_c6d9ef131d394299abae730c3d467d74/haproxy/0.log"
Oct 30 19:41:14.466342 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:14.466313    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/alertmanager/0.log"
Oct 30 19:41:14.878754 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:14.878677    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/config-reloader/0.log"
Oct 30 19:41:15.287754 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:15.287717    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/kube-rbac-proxy-web/0.log"
Oct 30 19:41:15.697775 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:15.697749    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/kube-rbac-proxy/0.log"
Oct 30 19:41:16.110772 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:16.110673    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/kube-rbac-proxy-metric/0.log"
Oct 30 19:41:16.522554 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:16.522521    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/prom-label-proxy/0.log"
Oct 30 19:41:16.932844 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:16.932770    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_008b7f60-84ef-44fb-a059-66ea1da81563/init-config-reloader/0.log"
Oct 30 19:41:17.879365 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:17.879334    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r6mch_549292be-cfde-49a0-bd85-b1854a005bff/kube-state-metrics/0.log"
Oct 30 19:41:18.068079 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068047    5972 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"]
Oct 30 19:41:18.068499 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068372    5972 cpu_manager.go:414] "containerMap: removing container" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="copy"
Oct 30 19:41:18.068499 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068382    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="copy"
Oct 30 19:41:18.068499 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068398    5972 cpu_manager.go:414] "containerMap: removing container" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="gather"
Oct 30 19:41:18.068499 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068403    5972 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="gather"
Oct 30 19:41:18.068499 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068450    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="copy"
Oct 30 19:41:18.068499 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.068460    5972 memory_manager.go:365] "RemoveStaleState removing state" podUID="f728d41e-03d6-4181-9abb-f0a21ff2c26b" containerName="gather"
Oct 30 19:41:18.070321 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.070306    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.121335 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.121302    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zsclz\"/\"kube-root-ca.crt\""
Oct 30 19:41:18.123131 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.123105    5972 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zsclz\"/\"openshift-service-ca.crt\""
Oct 30 19:41:18.168941 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.168852    5972 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zsclz\"/\"default-dockercfg-cwfbv\""
Oct 30 19:41:18.169711 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.169692    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"]
Oct 30 19:41:18.180985 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.180954    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-proc\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.181173 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.180998    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-sys\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.181173 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.181076    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-lib-modules\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.181173 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.181151    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsdx\" (UniqueName: \"kubernetes.io/projected/7f80d593-9e31-4555-95c6-efb232dee1e3-kube-api-access-ffsdx\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.181327 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.181233    5972 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-podres\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282299 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282251    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-podres\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282322    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-proc\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282351    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-sys\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282372    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-lib-modules\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282398    5972 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsdx\" (UniqueName: \"kubernetes.io/projected/7f80d593-9e31-4555-95c6-efb232dee1e3-kube-api-access-ffsdx\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282402    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-podres\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282457    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-proc\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282477    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-sys\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.282574 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.282518    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f80d593-9e31-4555-95c6-efb232dee1e3-lib-modules\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.290895 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.290857    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r6mch_549292be-cfde-49a0-bd85-b1854a005bff/kube-rbac-proxy-main/0.log"
Oct 30 19:41:18.342774 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.342745    5972 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsdx\" (UniqueName: \"kubernetes.io/projected/7f80d593-9e31-4555-95c6-efb232dee1e3-kube-api-access-ffsdx\") pod \"perf-node-gather-daemonset-bvrbl\" (UID: \"7f80d593-9e31-4555-95c6-efb232dee1e3\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.381874 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.381827    5972 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:18.703144 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.703115    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r6mch_549292be-cfde-49a0-bd85-b1854a005bff/kube-rbac-proxy-self/0.log"
Oct 30 19:41:18.748748 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:18.748709    5972 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"]
Oct 30 19:41:18.752451 ip-10-0-134-133 kubenswrapper[5972]: W1030 19:41:18.752420    5972 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f80d593_9e31_4555_95c6_efb232dee1e3.slice/crio-5d3e2027a9f5640de214566ab49f2809cb4f2d94ca4ca0dac93f16b16e8cfb6e WatchSource:0}: Error finding container 5d3e2027a9f5640de214566ab49f2809cb4f2d94ca4ca0dac93f16b16e8cfb6e: Status 404 returned error can't find the container with id 5d3e2027a9f5640de214566ab49f2809cb4f2d94ca4ca0dac93f16b16e8cfb6e
Oct 30 19:41:19.122934 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:19.122860    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-58f458b77f-264xl_166c00de-9f0d-49c7-8cd8-37c99b9af0a7/metrics-server/0.log"
Oct 30 19:41:19.394034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:19.393945    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl" event={"ID":"7f80d593-9e31-4555-95c6-efb232dee1e3","Type":"ContainerStarted","Data":"211c697ce766e0f5ab8911d6e778854ff6507c702ebf943768f5e6f21e8b3fe0"}
Oct 30 19:41:19.394034 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:19.394001    5972 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl" event={"ID":"7f80d593-9e31-4555-95c6-efb232dee1e3","Type":"ContainerStarted","Data":"5d3e2027a9f5640de214566ab49f2809cb4f2d94ca4ca0dac93f16b16e8cfb6e"}
Oct 30 19:41:19.394223 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:19.394083    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:19.506603 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:19.506519    5972 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl" podStartSLOduration=1.506504773 podStartE2EDuration="1.506504773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:19.50610429 +0000 UTC m=+3729.298071577" watchObservedRunningTime="2026-10-30 19:41:19.506504773 +0000 UTC m=+3729.298472039"
Oct 30 19:41:19.951080 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:19.951050    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49tz6_bfbb88e7-35a1-4d40-9365-66a6ba00adb9/node-exporter/0.log"
Oct 30 19:41:20.360942 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:20.360856    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49tz6_bfbb88e7-35a1-4d40-9365-66a6ba00adb9/kube-rbac-proxy/0.log"
Oct 30 19:41:20.770872 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:20.770846    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49tz6_bfbb88e7-35a1-4d40-9365-66a6ba00adb9/init-textfile/0.log"
Oct 30 19:41:22.875185 ip-10-0-134-133 kubenswrapper[5972]: E1030 19:41:22.875134    5972 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pk298" containerName="extractor"
Oct 30 19:41:23.659376 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:23.659334    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gr5bl_cf611fab-4203-425e-8db6-6ab6072ec4c2/kube-rbac-proxy-main/0.log"
Oct 30 19:41:24.074464 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:24.074437    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gr5bl_cf611fab-4203-425e-8db6-6ab6072ec4c2/kube-rbac-proxy-self/0.log"
Oct 30 19:41:24.489444 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:24.489412    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gr5bl_cf611fab-4203-425e-8db6-6ab6072ec4c2/openshift-state-metrics/0.log"
Oct 30 19:41:24.927720 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:24.927687    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/prometheus/0.log"
Oct 30 19:41:25.330415 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:25.330386    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/config-reloader/0.log"
Oct 30 19:41:25.409455 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:25.409427    5972 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-bvrbl"
Oct 30 19:41:25.744032 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:25.743990    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/thanos-sidecar/0.log"
Oct 30 19:41:26.154382 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:26.154297    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/kube-rbac-proxy-web/0.log"
Oct 30 19:41:26.565043 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:26.565014    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/kube-rbac-proxy/0.log"
Oct 30 19:41:26.975249 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:26.975210    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/kube-rbac-proxy-thanos/0.log"
Oct 30 19:41:27.388040 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:27.387964    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e390c06-9942-45ab-b31e-ff7f21ad2993/init-config-reloader/0.log"
Oct 30 19:41:27.806508 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:27.806469    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mpc5m_522412d0-7818-4a3d-975f-c816eb5d7a13/prometheus-operator/0.log"
Oct 30 19:41:28.212628 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:28.212602    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mpc5m_522412d0-7818-4a3d-975f-c816eb5d7a13/kube-rbac-proxy/0.log"
Oct 30 19:41:29.050274 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:29.050243    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9b8d4dbf-wn9xw_f768f332-b6fe-48bb-8e63-ea57b00f0b7d/telemeter-client/0.log"
Oct 30 19:41:29.460889 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:29.460858    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9b8d4dbf-wn9xw_f768f332-b6fe-48bb-8e63-ea57b00f0b7d/reload/0.log"
Oct 30 19:41:29.873311 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:29.873226    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9b8d4dbf-wn9xw_f768f332-b6fe-48bb-8e63-ea57b00f0b7d/kube-rbac-proxy/0.log"
Oct 30 19:41:30.289830 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:30.289804    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c948665c7-79g5z_65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/thanos-query/0.log"
Oct 30 19:41:30.702105 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:30.702074    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c948665c7-79g5z_65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/kube-rbac-proxy-web/0.log"
Oct 30 19:41:31.116322 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:31.116237    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c948665c7-79g5z_65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/kube-rbac-proxy/0.log"
Oct 30 19:41:31.529634 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:31.529602    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c948665c7-79g5z_65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/prom-label-proxy/0.log"
Oct 30 19:41:31.949142 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:31.949113    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c948665c7-79g5z_65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/kube-rbac-proxy-rules/0.log"
Oct 30 19:41:32.358805 ip-10-0-134-133 kubenswrapper[5972]: I1030 19:41:32.358711    5972 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c948665c7-79g5z_65e30ed0-eb80-40bc-87bb-1afbdda8f1e3/kube-rbac-proxy-metrics/0.log"