Nov 02 17:37:22.735122 ip-10-0-141-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 17:37:22.735370 ip-10-0-141-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 17:37:22.735475 ip-10-0-141-84 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 17:37:22.735886 ip-10-0-141-84 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 17:37:32.984104 ip-10-0-141-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 17:37:32.984122 ip-10-0-141-84 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8c8807ddfe5648f0bfcce702f22a8c62 -- Nov 02 17:39:23.823283 ip-10-0-141-84 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:39:24.261604 ip-10-0-141-84 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:39:24.261604 ip-10-0-141-84 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:39:24.261604 ip-10-0-141-84 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:39:24.261604 ip-10-0-141-84 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:39:24.265974 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.265893 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.265974 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.265976 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.265982 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.265985 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266008 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266012 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266015 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266018 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266021 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266023 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266026 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266028 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266031 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266033 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266036 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266039 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266041 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266044 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266046 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266049 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266051 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.266058 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266054 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266056 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266059 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266061 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266064 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266067 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266070 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266073 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266075 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266078 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266081 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266083 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266086 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266088 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266092 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266094 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266097 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266100 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266102 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266105 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.266522 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266109 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266111 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266113 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266116 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266118 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266121 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266123 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266126 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266128 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266131 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266133 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266136 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266139 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266141 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266145 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266147 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266150 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266153 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266155 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266158 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.267048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266160 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266163 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266165 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266168 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266170 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266173 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266175 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266177 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266180 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266183 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266186 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266188 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266191 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266193 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266197 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266201 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266204 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266207 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266210 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.267613 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266212 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266215 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266218 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266220 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266222 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266225 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266228 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266230 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266233 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266236 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266238 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266241 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266243 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266246 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266248 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266251 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266254 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266256 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266259 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.268119 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266261 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266264 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266266 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266268 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266272 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266276 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.266279 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268165 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268173 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268177 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268180 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268184 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268188 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268191 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268194 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268196 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268199 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268201 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268204 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.268566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268208 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268212 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268215 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268218 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268221 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268224 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268226 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268229 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268231 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268234 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268236 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268239 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268241 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268244 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268246 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268249 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268252 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268255 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268258 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268261 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.269048 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268263 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268266 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268268 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268271 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268274 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268276 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268279 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268281 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268284 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268286 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268289 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268291 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268294 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268296 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268299 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268302 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268304 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268307 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268309 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.269531 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268312 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268314 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268317 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268319 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268322 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268325 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268327 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268329 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268332 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268335 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268338 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268340 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268343 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268346 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268348 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268351 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268354 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268356 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268359 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.270008 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268361 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268364 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268366 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268369 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268371 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268374 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268376 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268379 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268381 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268384 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268387 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268389 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268392 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268394 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268397 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268399 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268401 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268404 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268407 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268409 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.270702 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268411 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268415 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268417 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268420 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268423 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268426 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268431 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268436 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268439 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268442 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268445 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268448 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268450 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268453 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268455 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268458 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268932 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268937 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268940 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.271291 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268942 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268945 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268947 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268950 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268952 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268955 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268957 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268960 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268962 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268965 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268967 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268970 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268972 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268975 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268977 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268981 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268984 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.268987 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269004 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269008 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.271739 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269012 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269015 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269017 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269020 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269022 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269025 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269027 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269030 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269033 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269037 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269042 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269045 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269048 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269051 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269054 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269056 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269059 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269062 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269065 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.272223 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269067 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269070 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269072 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269075 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269077 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269080 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269083 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269085 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269088 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269091 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269093 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269096 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269098 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269102 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269104 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269107 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269109 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269112 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269114 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269117 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.272668 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269119 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269122 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269124 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269127 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269129 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269132 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269134 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269137 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269139 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269142 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269145 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269147 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269150 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269152 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269155 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269157 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269160 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269163 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269165 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.273148 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269168 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269170 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269173 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269176 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269178 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269181 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269183 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269186 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269189 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269191 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269193 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269196 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269198 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269201 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269203 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269206 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269208 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269211 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269213 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.273593 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269215 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269218 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269221 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269223 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269225 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.269229 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269324 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269332 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269338 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269343 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269347 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269351 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269355 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269359 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269363 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269366 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269369 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269373 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269376 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269379 2581 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269382 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269386 2581 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269389 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:39:24.274072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269392 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269396 2581 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269398 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269402 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269405 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269408 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269413 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269416 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269422 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269425 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269428 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269431 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269434 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269437 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269440 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269444 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269448 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269451 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269454 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269457 2581 flags.go:64] FLAG: --enable-server="true" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269461 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269465 2581 flags.go:64] FLAG: --event-burst="100" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269468 2581 flags.go:64] FLAG: --event-qps="50" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269471 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:39:24.274607 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269474 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269477 2581 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269481 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269484 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269487 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269490 2581 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269493 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269497 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269500 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269504 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269507 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269509 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269512 2581 flags.go:64] FLAG: --feature-gates="" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269516 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269519 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269523 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269527 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269530 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269534 2581 flags.go:64] FLAG: --help="false" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269537 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269540 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269544 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269546 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:39:24.275258 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269550 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269553 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269556 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269559 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269564 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269567 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269569 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269573 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269576 2581 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269579 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269582 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269585 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269588 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269591 2581 flags.go:64] FLAG: --lock-file="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269593 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269596 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269599 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269606 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269609 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269612 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269614 2581 flags.go:64] FLAG: --logging-format="text" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269617 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269621 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269624 2581 flags.go:64] FLAG: --manifest-url="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269627 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:39:24.275813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269632 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269636 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269639 2581 flags.go:64] FLAG: --max-pods="110" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269643 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269646 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269649 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269652 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269655 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269658 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269661 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269668 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269673 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269676 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269679 2581 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269682 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269685 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269688 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269691 2581 flags.go:64] FLAG: --port="10250" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269694 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269697 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0e8575bfc32139708" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269700 2581 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269703 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269707 2581 flags.go:64] FLAG: --register-node="true" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269710 2581 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269714 2581 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269717 2581 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:39:24.276431 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269720 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269723 2581 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269727 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269731 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269734 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269737 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269741 2581 flags.go:64] FLAG: --runonce="false" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269744 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269748 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269751 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269754 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269756 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269760 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269763 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269766 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269769 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269772 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269776 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269779 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269782 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269785 2581 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269788 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269794 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269797 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269799 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:39:24.277069 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269803 2581 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269806 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269809 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269812 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269815 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269818 2581 flags.go:64] FLAG: --v="2" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269822 2581 flags.go:64] FLAG: --version="false" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269827 2581 flags.go:64] FLAG: --vmodule="" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269831 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.269834 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271139 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271151 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271155 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271161 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271164 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271167 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271173 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271176 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271179 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271181 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271184 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271188 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.277651 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271193 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271196 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271200 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271203 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271207 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271210 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271212 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271218 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271221 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271224 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271226 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271229 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271232 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271235 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271237 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271240 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271243 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271245 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271249 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.278188 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271254 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271258 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271260 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271263 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271268 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271271 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271273 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271276 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271279 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271281 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271284 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271286 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271289 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271295 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271298 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271302 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271305 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271308 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271312 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.278639 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271316 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271319 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271322 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271324 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271327 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271330 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271335 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271338 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271341 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271343 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271346 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271349 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271351 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271355 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271358 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271360 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271363 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271367 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271370 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271375 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.279178 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271378 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271380 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271383 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271386 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271388 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271391 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271394 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271396 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271400 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271403 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271406 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271411 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271414 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271416 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271419 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271422 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271425 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271427 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271430 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271433 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.279656 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271435 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271438 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271441 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271443 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271448 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271451 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271454 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271457 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271459 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271463 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271465 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271468 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271471 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271474 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271476 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271479 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.280179 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.271517 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271590 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271597 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271601 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271603 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271607 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271610 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271613 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271616 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271619 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271622 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271624 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271627 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271630 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271635 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271638 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271641 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271644 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271646 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.280542 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271649 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271652 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271654 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271657 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271663 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271666 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271670 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271672 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271678 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271680 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271683 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271686 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271692 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271696 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271699 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271702 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271705 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271709 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.280984 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271712 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271718 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271720 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271723 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271726 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271728 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271731 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271733 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271736 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271739 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271742 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271745 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271747 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271750 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271756 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271760 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271763 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271765 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271769 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271772 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.281434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271775 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271777 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271780 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271783 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271786 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271788 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271794 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271797 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271799 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271802 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271805 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271808 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271811 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271813 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271816 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271819 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271822 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271824 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271827 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271832 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.281905 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271835 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271837 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271840 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271843 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271845 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271848 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271851 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271853 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271856 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271858 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271861 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271867 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271870 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271873 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271875 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271878 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271881 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271883 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271886 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271889 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.282400 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271891 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271894 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271898 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271902 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271908 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271910 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271913 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271917 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.271920 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.279254 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.279269 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.279283 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.279287 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279345 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279349 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279352 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279355 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279358 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279361 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.282872 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279363 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279366 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279369 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279372 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279375 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279378 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279380 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279383 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279386 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279389 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279392 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279394 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279397 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279400 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279402 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279405 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279408 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279411 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279414 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279416 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.283434 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279419 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279422 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279424 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279428 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279430 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279433 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279435 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279438 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279441 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279443 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279447 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279451 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279454 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279458 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279461 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279463 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279466 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279469 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279472 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.283893 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279474 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279477 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279480 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279482 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279485 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279488 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279493 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279497 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279503 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279509 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279514 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279517 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279520 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279522 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279525 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279528 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279532 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279534 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279537 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.284350 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279540 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279543 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279546 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279548 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279551 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279553 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279556 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279558 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279561 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279563 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279566 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279569 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279571 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279574 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279576 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279579 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279582 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279584 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279587 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279589 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.284820 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279592 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279594 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279597 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279599 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279602 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279604 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279607 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279609 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279612 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279615 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279617 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279620 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279623 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279626 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279629 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279631 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279634 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279636 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279639 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279641 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.285343 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279644 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279647 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.279675 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279838 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279843 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279846 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279849 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279852 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279854 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279857 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279860 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279862 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279865 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279868 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279872 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.285816 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279875 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279878 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279881 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279884 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279887 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279889 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279892 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279895 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279897 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279899 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279903 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279905 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279908 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279910 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279913 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279916 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279919 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279921 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279924 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279926 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.286207 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279929 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279931 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279934 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279937 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279939 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279942 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279944 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279947 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279949 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279952 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279955 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279957 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279960 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279962 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279964 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279967 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279969 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279972 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279974 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.286682 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279977 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279980 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279982 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279985 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.279988 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280004 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280009 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280013 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280016 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280019 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280022 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280025 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280027 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280030 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280032 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280035 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280037 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280040 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280042 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280045 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.287217 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280047 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280050 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280052 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280055 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280058 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280060 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280063 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280066 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280068 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280071 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280073 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280076 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280078 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280081 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280084 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280087 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280090 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280093 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280095 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280098 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.287794 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280101 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280103 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280106 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280110 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280113 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280116 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280118 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280121 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280124 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280126 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280129 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280131 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280134 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280136 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280141 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.288419 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.280166 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280199 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280204 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280208 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280211 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280215 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280218 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280222 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280224 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280227 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280230 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280232 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280235 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280238 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280240 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280246 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280249 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280252 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280255 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280257 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:24.288772 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280260 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280262 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280265 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280267 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280270 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280272 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280275 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280278 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280280 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280283 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280286 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280288 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280292 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280295 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280297 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280300 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280303 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280307 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280309 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:24.289263 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280312 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280315 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280317 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280320 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280322 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280325 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280327 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280330 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280332 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280335 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280338 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280341 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280343 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280346 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280348 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280351 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280353 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280356 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280358 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280361 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:24.289734 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280363 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280366 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280369 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280371 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280374 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280377 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280380 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280383 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280386 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280389 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280392 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280395 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280397 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280400 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280402 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280405 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280407 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280410 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280413 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280415 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:24.290398 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280418 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280420 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280423 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280426 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280430 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280434 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280437 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280440 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280442 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280446 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280448 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280451 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280453 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280456 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280459 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280461 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280464 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280467 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280470 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:24.290922 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280473 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280475 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280478 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280482 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280485 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280487 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280490 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280492 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:24.280497 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.280554 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.282907 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.284016 2581 server.go:1014] "Starting client certificate rotation" Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.284108 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.284141 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.286245 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:39:24.291407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.287410 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:39:24.297072 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.297057 2581 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:39:24.302721 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.302702 2581 log.go:25] "Validated CRI v1 image API" Nov 02 17:39:24.303643 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.303614 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:39:24.309903 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.309883 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9ef89d6b-bd4e-4ff3-afcf-d988820f4553:/dev/nvme0n1p3 cfdb2b64-95f8-4df4-80e2-4db4600b2d98:/dev/nvme0n1p4] Nov 02 17:39:24.309974 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.309901 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:39:24.314184 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.314164 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:39:24.315133 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.315017 2581 manager.go:217] Machine: {Timestamp:2026-11-02 17:39:24.313615854 +0000 UTC m=+0.382330208 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096440 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae62e3970f6bdcbba9a13a2e3beba SystemUUID:ec2ae62e-3970-f6bd-cbba-9a13a2e3beba BootID:8c8807dd-fe56-48f0-bfcc-e702f22a8c62 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:67:68:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:67:68:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:7d:34:6c:be:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:39:24.315133 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.315132 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:39:24.315226 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.315216 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:39:24.316408 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.316382 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:39:24.316543 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.316411 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-84.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:39:24.316583 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.316545 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:39:24.316583 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.316553 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:39:24.316583 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.316569 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:39:24.316654 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.316584 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:39:24.317815 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.317805 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:39:24.322274 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322261 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:39:24.322551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322392 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:39:24.322551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322423 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:39:24.322551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322440 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:39:24.322551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322451 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:39:24.322551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322461 2581 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:39:24.322551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322468 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:39:24.322826 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.322810 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:39:24.325413 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.325396 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:39:24.332091 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.332076 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:39:24.333083 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333068 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:39:24.333170 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333108 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:39:24.333298 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333285 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:39:24.333357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333308 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:39:24.333357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333319 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:39:24.333357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333329 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:39:24.333357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333338 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:39:24.333357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.333347 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:39:24.334026 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.334016 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:39:24.334075 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.334030 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:39:24.334075 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.334041 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:39:24.334075 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.334051 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:39:24.334279 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.334077 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:39:24.334279 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.334095 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:39:24.335548 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.335526 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:39:24.335548 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.335552 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:39:24.339497 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.339482 2581 server.go:1263] "Started kubelet" Nov 02 17:39:24.339573 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.339543 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:39:24.340289 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.340237 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:39:24.340375 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.340309 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:39:24.344377 ip-10-0-141-84 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:39:24.344783 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.344393 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:39:24.345728 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.345707 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:39:24.347763 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.347742 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-84.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 17:39:24.349325 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.349298 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-84.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 17:39:24.349407 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.349386 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:39:24.351586 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.351564 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 17:39:24.352741 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.351542 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-84.us-east-2.compute.internal.18e44b22c6f8a37a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-84.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-84.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:24.339454842 +0000 UTC m=+0.408169195,LastTimestamp:2026-11-02 17:39:24.339454842 +0000 UTC m=+0.408169195,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-84.us-east-2.compute.internal,}" Nov 02 17:39:24.355681 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.355664 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:39:24.355763 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.355688 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:39:24.355763 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.355753 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.356404 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356383 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:39:24.356404 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356386 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:39:24.356558 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356409 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:39:24.356558 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356512 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:39:24.356558 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356521 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:39:24.356686 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356668 2581 factory.go:153] Registering CRI-O factory Nov 02 17:39:24.356753 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356690 2581 factory.go:223] Registration of the crio container factory successfully Nov 02 17:39:24.356753 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356743 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:39:24.356753 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356752 2581 factory.go:55] Registering systemd factory Nov 02 17:39:24.356853 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356760 2581 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:39:24.356853 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356778 2581 factory.go:103] Registering Raw factory Nov 02 17:39:24.356853 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.356792 2581 manager.go:1199] Started watching for new ooms in manager Nov 02 17:39:24.357761 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.357750 2581 manager.go:319] Starting recovery of all containers Nov 02 17:39:24.358061 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.358036 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-84.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 17:39:24.362277 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.362247 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5tngz" Nov 02 17:39:24.368516 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.368499 2581 manager.go:324] Recovery completed Nov 02 17:39:24.370150 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.370130 2581 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 17:39:24.374158 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.374133 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 17:39:24.374569 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.374552 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.376845 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.376822 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.376916 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.376865 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.376916 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.376880 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.377544 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.377527 2581 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:39:24.377544 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.377542 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:39:24.377701 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.377549 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.377701 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.377561 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:39:24.378216 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.378196 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5tngz" Nov 02 17:39:24.378419 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.378404 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.378463 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.378435 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.378463 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.378449 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.379749 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.379735 2581 policy_none.go:50] "Start" Nov 02 17:39:24.379811 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.379759 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:39:24.379811 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.379773 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:39:24.381813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.381798 2581 policy_none.go:44] "Start" Nov 02 17:39:24.414593 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.414559 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:39:24.417225 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417211 2581 manager.go:341] "Starting Device Plugin manager" Nov 02 17:39:24.417329 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.417249 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:39:24.417329 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417260 2581 server.go:87] "Starting device plugin registration server" Nov 02 17:39:24.417548 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417536 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:39:24.417582 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417551 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:39:24.417678 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417663 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:39:24.417786 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417775 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:39:24.417786 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.417785 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:39:24.418528 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.418511 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:39:24.418607 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.418560 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.518272 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.518220 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.520007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.519225 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.520007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.519256 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.520007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.519270 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.520007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.519304 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.530388 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.530367 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.530432 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.530397 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-84.us-east-2.compute.internal\": node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.551316 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.551282 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.554585 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.554558 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:39:24.555778 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.555757 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:39:24.555874 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.555784 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:39:24.555874 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.555805 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:39:24.555874 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.555844 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 17:39:24.566062 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.565906 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:24.652088 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.652040 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.656230 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.656208 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal"] Nov 02 17:39:24.656314 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.656300 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.656752 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.656730 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.656817 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.656759 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.657361 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.657346 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.657440 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.657379 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.657440 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.657398 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.659625 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.659608 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.659786 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.659770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.659842 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.659800 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.660359 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.660345 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.660435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.660374 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.660435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.660385 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.660435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.660346 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.660541 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.660439 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.660541 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.660450 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.662344 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.662329 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.662423 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.662354 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.663077 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.663060 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.663150 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.663088 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.663150 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.663106 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.679136 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.679111 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.681538 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.681523 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.752480 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.752435 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.757122 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.757102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.757238 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.757138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.757238 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.757202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.757238 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.757202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.852747 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.852664 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.857968 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.857949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07e33052aa2d73769dbf3107e42483d7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"07e33052aa2d73769dbf3107e42483d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.953139 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:24.953109 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:24.958467 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.958446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07e33052aa2d73769dbf3107e42483d7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"07e33052aa2d73769dbf3107e42483d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.958525 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.958498 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07e33052aa2d73769dbf3107e42483d7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"07e33052aa2d73769dbf3107e42483d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.982626 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.982591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:24.984739 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:24.984724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:26.334845 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:26.334819 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:26.435369 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:26.435295 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:26.535393 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:26.535367 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-east-2.compute.internal\" not found" Nov 02 17:39:26.538963 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.538940 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:26.565059 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.565025 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:39:26.565468 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.565176 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:39:26.565468 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.565209 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:39:26.620883 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.620857 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:26.637698 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.637677 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:26.637790 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.637707 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:39:26.639344 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.639330 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:26.650005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.649989 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:39:26.651753 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.651733 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:39:26.655042 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.655029 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:39:26.662140 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.662125 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:39:26.665320 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.665275 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:34:25 +0000 UTC" deadline="2028-04-06 10:38:29.225946394 +0000 UTC" Nov 02 17:39:26.665382 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.665320 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12496h59m2.560628651s" Nov 02 17:39:26.728842 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.728776 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vjlq5" Nov 02 17:39:26.739464 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.739446 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vjlq5" Nov 02 17:39:26.902566 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:26.902534 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4115a2f4c3236862e1ddc68e0eda82.slice/crio-7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d WatchSource:0}: Error finding container 7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d: Status 404 returned error can't find the container with id 7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d Nov 02 17:39:26.903103 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:26.903083 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e33052aa2d73769dbf3107e42483d7.slice/crio-57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b WatchSource:0}: Error finding container 57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b: Status 404 returned error can't find the container with id 57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b Nov 02 17:39:26.908727 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:26.908703 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:39:27.604857 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.604822 2581 apiserver.go:52] "Watching apiserver" Nov 02 17:39:27.615220 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.615193 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:39:27.615525 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.615505 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-nvj7s","kube-system/global-pull-secret-syncer-xmv2b","openshift-image-registry/node-ca-25nk6","openshift-multus/network-metrics-daemon-rhqwb","openshift-multus/multus-hqxnx","kube-system/konnectivity-agent-rkl7h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97","openshift-cluster-node-tuning-operator/tuned-dbxfx","openshift-dns/node-resolver-zb589","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-zw4rm","openshift-network-diagnostics/network-check-target-89pdp","openshift-network-operator/iptables-alerter-246zz"] Nov 02 17:39:27.618098 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.618071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.621471 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.621443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:27.621573 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.621517 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:27.622413 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.622300 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.622413 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.622314 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.622413 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.622387 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:39:27.622582 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.622417 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjwn5\"" Nov 02 17:39:27.626407 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.625532 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.627818 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.627794 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:27.627926 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.627874 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:27.630038 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.630015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.634288 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.634268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.638723 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.638704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.641149 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.641128 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.647995 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.647975 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.648113 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.647972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-socket-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.648113 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.648072 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:27.648113 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7f4\" (UniqueName: \"kubernetes.io/projected/12700290-2871-451f-9f86-3ea2830de15d-kube-api-access-2p7f4\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.648319 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-socket-dir-parent\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648319 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee56fd64-8479-45f4-b098-16208b7ced9f-tmp-dir\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.648319 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648203 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.648319 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-device-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.648319 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-sys-fs\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.648319 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648283 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12700290-2871-451f-9f86-3ea2830de15d-host\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.648613 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-system-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648613 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-multus-certs\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648613 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-etc-kubernetes\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648613 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20c5900d-985b-4db3-8232-5c24ff85cc01-agent-certs\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.648613 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4n6r\" (UniqueName: \"kubernetes.io/projected/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kube-api-access-b4n6r\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.648835 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648835 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-k8s-cni-cncf-io\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648835 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-bin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.648835 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:27.648835 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflxg\" (UniqueName: \"kubernetes.io/projected/ee56fd64-8479-45f4-b098-16208b7ced9f-kube-api-access-zflxg\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.649070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.649070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cnibin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-hostroot\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.648984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-daemon-config\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20c5900d-985b-4db3-8232-5c24ff85cc01-konnectivity-ca\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.649350 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee56fd64-8479-45f4-b098-16208b7ced9f-hosts-file\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.649600 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-kubelet\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649713 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmqr\" (UniqueName: \"kubernetes.io/projected/8271b561-5608-4f0c-93eb-0850dcdcd9c6-kube-api-access-gqmqr\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649713 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12700290-2871-451f-9f86-3ea2830de15d-serviceca\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.649824 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-multus\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649824 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-conf-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.649928 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdgd\" (UniqueName: \"kubernetes.io/projected/9e998028-a165-4d2b-8c14-c422d8a4d099-kube-api-access-lkdgd\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:27.649928 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-registration-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.649928 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cni-binary-copy\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.650082 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.649939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-netns\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.650140 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.650115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:27.651581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.650854 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.653741 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.653711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.656984 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.656633 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.673127 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.673102 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:27.723387 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.723288 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.723387 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.723357 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:39:27.724647 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.724623 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.724746 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.724691 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:39:27.724837 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.724823 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.725005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.724868 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:39:27.725005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.724944 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zqcnj\"" Nov 02 17:39:27.725005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.724981 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:39:27.725162 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725137 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.725212 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725195 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b2827\"" Nov 02 17:39:27.725264 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725234 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:39:27.725312 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725280 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.725356 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725324 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.725398 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725370 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.725455 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725444 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:39:27.725534 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725522 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.725586 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.725554 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726483 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726555 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpgl9\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726555 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726578 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726622 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726633 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4dl6t\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726634 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726670 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726696 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kkqh6\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726697 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzjcr\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726816 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:39:27.726970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.726945 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zvsl\"" Nov 02 17:39:27.728189 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.728153 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:39:27.728534 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.728498 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jk79z\"" Nov 02 17:39:27.728716 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.728700 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:39:27.739135 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.739110 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:39:27.742455 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.742421 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:34:26 +0000 UTC" deadline="2028-05-28 08:36:45.038529702 +0000 UTC" Nov 02 17:39:27.742455 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.742451 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13742h57m17.29608243s" Nov 02 17:39:27.750727 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cni-binary-copy\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.750903 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-ovn\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.750903 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-netd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.750903 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6zm\" (UniqueName: \"kubernetes.io/projected/68c1bf64-b418-49c2-9753-4fad4bff08ae-kube-api-access-2f6zm\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.750903 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-var-lib-kubelet\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.751189 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-socket-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.751189 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.750995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee56fd64-8479-45f4-b098-16208b7ced9f-tmp-dir\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.751189 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-script-lib\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751189 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-systemd\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.751189 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwnb\" (UniqueName: \"kubernetes.io/projected/094f3261-e65e-4fd0-893d-0e9c0f99988f-kube-api-access-pxwnb\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-sys-fs\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20c5900d-985b-4db3-8232-5c24ff85cc01-agent-certs\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-slash\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751322 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-node-log\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-k8s-cni-cncf-io\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-dbus\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.751435 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751431 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68c1bf64-b418-49c2-9753-4fad4bff08ae-host-slash\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751508 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cni-binary-copy\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92r8b\" (UniqueName: \"kubernetes.io/projected/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-kube-api-access-92r8b\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751698 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-daemon-config\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20c5900d-985b-4db3-8232-5c24ff85cc01-konnectivity-ca\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee56fd64-8479-45f4-b098-16208b7ced9f-hosts-file\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-env-overrides\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751868 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee56fd64-8479-45f4-b098-16208b7ced9f-tmp-dir\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pr69\" (UniqueName: \"kubernetes.io/projected/98c18ffc-40f4-4360-9095-b08f95cab47c-kube-api-access-9pr69\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.751941 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.751966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-cnibin\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752061 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-conf-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdgd\" (UniqueName: \"kubernetes.io/projected/9e998028-a165-4d2b-8c14-c422d8a4d099-kube-api-access-lkdgd\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-kubelet\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-bin\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-modprobe-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysconfig\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-registration-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-netns\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752520 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-daemon-config\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-netns\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.752581 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752565 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-etc-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-log-socket\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752617 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-kubernetes\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7f4\" (UniqueName: \"kubernetes.io/projected/12700290-2871-451f-9f86-3ea2830de15d-kube-api-access-2p7f4\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-socket-dir-parent\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752700 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-config\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-device-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-var-lib-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68c1bf64-b418-49c2-9753-4fad4bff08ae-iptables-alerter-script\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752916 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-conf\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12700290-2871-451f-9f86-3ea2830de15d-host\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.752968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-system-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20c5900d-985b-4db3-8232-5c24ff85cc01-konnectivity-ca\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-multus-certs\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753253 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-conf-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-etc-kubernetes\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.753449 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-systemd-units\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-run\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-host\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4n6r\" (UniqueName: \"kubernetes.io/projected/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kube-api-access-b4n6r\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-bin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflxg\" (UniqueName: \"kubernetes.io/projected/ee56fd64-8479-45f4-b098-16208b7ced9f-kube-api-access-zflxg\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-tmp\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-registration-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cnibin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-hostroot\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-netns\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-kubelet-config\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753759 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-lib-modules\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-tuned\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.754322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-kubelet\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmqr\" (UniqueName: \"kubernetes.io/projected/8271b561-5608-4f0c-93eb-0850dcdcd9c6-kube-api-access-gqmqr\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-systemd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753955 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98c18ffc-40f4-4360-9095-b08f95cab47c-ovn-node-metrics-cert\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-sys\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12700290-2871-451f-9f86-3ea2830de15d-serviceca\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-multus\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754139 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-socket-dir-parent\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-multus\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754268 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-device-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754306 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12700290-2871-451f-9f86-3ea2830de15d-host\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754306 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-system-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754414 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-socket-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754465 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-bin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754477 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754527 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-sys-fs\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.755192 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.754812 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.753759 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee56fd64-8479-45f4-b098-16208b7ced9f-hosts-file\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.754962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755003 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cnibin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755030 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-hostroot\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-multus-certs\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755256 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-etc-kubernetes\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755340 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-kubelet\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755340 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12700290-2871-451f-9f86-3ea2830de15d-serviceca\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.755466 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:28.25544052 +0000 UTC m=+3.043100223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755521 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:39:27.755841 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.755553 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-k8s-cni-cncf-io\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.759662 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.759638 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20c5900d-985b-4db3-8232-5c24ff85cc01-agent-certs\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.767553 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.767212 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:27.767553 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.767252 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:27.767553 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.767271 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:27.767553 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.767352 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:28.267330853 +0000 UTC m=+3.054990556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:27.769893 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.769818 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdgd\" (UniqueName: \"kubernetes.io/projected/9e998028-a165-4d2b-8c14-c422d8a4d099-kube-api-access-lkdgd\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:27.770453 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.770432 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4n6r\" (UniqueName: \"kubernetes.io/projected/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kube-api-access-b4n6r\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.770789 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.770773 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflxg\" (UniqueName: \"kubernetes.io/projected/ee56fd64-8479-45f4-b098-16208b7ced9f-kube-api-access-zflxg\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.771408 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.771386 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmqr\" (UniqueName: \"kubernetes.io/projected/8271b561-5608-4f0c-93eb-0850dcdcd9c6-kube-api-access-gqmqr\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.771617 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.771594 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7f4\" (UniqueName: \"kubernetes.io/projected/12700290-2871-451f-9f86-3ea2830de15d-kube-api-access-2p7f4\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.841586 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.841524 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"07e33052aa2d73769dbf3107e42483d7","Type":"ContainerStarted","Data":"57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b"} Nov 02 17:39:27.843206 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.843174 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"3f4115a2f4c3236862e1ddc68e0eda82","Type":"ContainerStarted","Data":"7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d"} Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-slash\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855302 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-node-log\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-dbus\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68c1bf64-b418-49c2-9753-4fad4bff08ae-host-slash\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855353 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-slash\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r8b\" (UniqueName: \"kubernetes.io/projected/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-kube-api-access-92r8b\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855419 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-dbus\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855439 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68c1bf64-b418-49c2-9753-4fad4bff08ae-host-slash\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855488 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-node-log\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-env-overrides\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pr69\" (UniqueName: \"kubernetes.io/projected/98c18ffc-40f4-4360-9095-b08f95cab47c-kube-api-access-9pr69\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856117 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-cnibin\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855640 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855640 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-cnibin\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855769 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-kubelet\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-bin\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-modprobe-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855881 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-bin\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysconfig\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.855980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-modprobe-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.856948 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-kubelet\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856131 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysconfig\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-netns\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856132 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856177 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-env-overrides\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856187 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-etc-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-netns\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-etc-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-log-socket\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-kubernetes\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856312 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-config\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856321 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-log-socket\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-var-lib-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856382 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-kubernetes\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68c1bf64-b418-49c2-9753-4fad4bff08ae-iptables-alerter-script\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-conf\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-systemd-units\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.857730 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-run\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-host\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-tmp\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-kubelet-config\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856604 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-lib-modules\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856682 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-run\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-tuned\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-systemd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856761 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-conf\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856784 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-config\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856790 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98c18ffc-40f4-4360-9095-b08f95cab47c-ovn-node-metrics-cert\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-var-lib-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856862 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-lib-modules\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-sys\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-systemd-units\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.856942 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-host\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858392 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.856949 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:27.857012 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:39:28.35699617 +0000 UTC m=+3.144656074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857123 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68c1bf64-b418-49c2-9753-4fad4bff08ae-iptables-alerter-script\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857191 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-sys\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-ovn\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-netd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6zm\" (UniqueName: \"kubernetes.io/projected/68c1bf64-b418-49c2-9753-4fad4bff08ae-kube-api-access-2f6zm\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-var-lib-kubelet\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-var-lib-kubelet\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857638 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-netd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857674 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-kubelet-config\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857685 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-ovn\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-systemd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-script-lib\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-systemd\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwnb\" (UniqueName: \"kubernetes.io/projected/094f3261-e65e-4fd0-893d-0e9c0f99988f-kube-api-access-pxwnb\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.858970 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857909 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-systemd\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.859712 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.857958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.859712 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.858023 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.859712 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.858407 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-script-lib\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.859712 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.859576 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-tmp\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.859712 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.859589 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-tuned\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.862143 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.862107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98c18ffc-40f4-4360-9095-b08f95cab47c-ovn-node-metrics-cert\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.872504 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.872456 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r8b\" (UniqueName: \"kubernetes.io/projected/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-kube-api-access-92r8b\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:27.873102 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.873078 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pr69\" (UniqueName: \"kubernetes.io/projected/98c18ffc-40f4-4360-9095-b08f95cab47c-kube-api-access-9pr69\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.874414 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.874394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6zm\" (UniqueName: \"kubernetes.io/projected/68c1bf64-b418-49c2-9753-4fad4bff08ae-kube-api-access-2f6zm\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.874873 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.874853 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwnb\" (UniqueName: \"kubernetes.io/projected/094f3261-e65e-4fd0-893d-0e9c0f99988f-kube-api-access-pxwnb\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:27.935726 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.935691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:39:27.942789 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.942762 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:39:27.946596 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:27.946532 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed1ac36_08cd_4ba5_9f21_74d8616dbdb1.slice/crio-315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12 WatchSource:0}: Error finding container 315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12: Status 404 returned error can't find the container with id 315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12 Nov 02 17:39:27.953384 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:27.953358 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12700290_2871_451f_9f86_3ea2830de15d.slice/crio-9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59 WatchSource:0}: Error finding container 9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59: Status 404 returned error can't find the container with id 9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59 Nov 02 17:39:27.955741 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.955586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hqxnx" Nov 02 17:39:27.964147 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.963963 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:27.965353 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:27.965326 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8271b561_5608_4f0c_93eb_0850dcdcd9c6.slice/crio-58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78 WatchSource:0}: Error finding container 58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78: Status 404 returned error can't find the container with id 58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78 Nov 02 17:39:27.970864 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.970843 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zb589" Nov 02 17:39:27.980764 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:27.980736 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee56fd64_8479_45f4_b098_16208b7ced9f.slice/crio-258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef WatchSource:0}: Error finding container 258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef: Status 404 returned error can't find the container with id 258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef Nov 02 17:39:27.992650 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.992620 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:39:27.996476 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.996454 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:27.998770 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:27.998749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:39:28.003707 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:28.003678 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c1bf64_b418_49c2_9753_4fad4bff08ae.slice/crio-7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e WatchSource:0}: Error finding container 7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e: Status 404 returned error can't find the container with id 7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e Nov 02 17:39:28.007266 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:28.007209 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c18ffc_40f4_4360_9095_b08f95cab47c.slice/crio-1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87 WatchSource:0}: Error finding container 1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87: Status 404 returned error can't find the container with id 1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87 Nov 02 17:39:28.016018 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.015991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:39:28.027091 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:39:28.027058 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094f3261_e65e_4fd0_893d_0e9c0f99988f.slice/crio-8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852 WatchSource:0}: Error finding container 8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852: Status 404 returned error can't find the container with id 8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852 Nov 02 17:39:28.260573 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.260474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:28.260700 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.260603 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:28.260700 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.260670 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:29.260657453 +0000 UTC m=+4.048317134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:28.365733 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.365566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:28.366014 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.365989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:28.366179 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.365756 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:28.366631 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.366161 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:28.366810 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.366798 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:39:29.366781083 +0000 UTC m=+4.154440778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:28.366915 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.366653 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:28.367012 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.366995 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:28.367162 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.367073 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:29.367042625 +0000 UTC m=+4.154702322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:28.743402 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.743316 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:34:26 +0000 UTC" deadline="2028-08-12 00:23:04.123814456 +0000 UTC" Nov 02 17:39:28.743402 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.743372 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15558h43m35.380446632s" Nov 02 17:39:28.837739 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.837704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:28.837906 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.837834 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:28.838331 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.838311 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:28.838485 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.838453 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:28.838549 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.838538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:28.838625 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:28.838607 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:28.855398 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.855129 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hqxnx" event={"ID":"8271b561-5608-4f0c-93eb-0850dcdcd9c6","Type":"ContainerStarted","Data":"58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78"} Nov 02 17:39:28.857302 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.857211 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" event={"ID":"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1","Type":"ContainerStarted","Data":"315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12"} Nov 02 17:39:28.866372 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.866340 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerStarted","Data":"8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852"} Nov 02 17:39:28.873924 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.873894 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" event={"ID":"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c","Type":"ContainerStarted","Data":"72d36d408eecc958e96ac0a39edef895bf8e85d8bbe01568507b7d6442b3bb0f"} Nov 02 17:39:28.890950 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.890912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkl7h" event={"ID":"20c5900d-985b-4db3-8232-5c24ff85cc01","Type":"ContainerStarted","Data":"e70cbc169df2a25beb706b74739c8e7a568a56c90863c60ad115517f354ed62d"} Nov 02 17:39:28.898193 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.898167 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-25nk6" event={"ID":"12700290-2871-451f-9f86-3ea2830de15d","Type":"ContainerStarted","Data":"9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59"} Nov 02 17:39:28.904201 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.904163 2581 generic.go:356] "Generic (PLEG): container finished" podID="3f4115a2f4c3236862e1ddc68e0eda82" containerID="27b70c98a073f1bb30277b3b1bdee1a016de81c46af55f5dc7e8346a1e39e36a" exitCode=0 Nov 02 17:39:28.904351 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.904258 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"3f4115a2f4c3236862e1ddc68e0eda82","Type":"ContainerDied","Data":"27b70c98a073f1bb30277b3b1bdee1a016de81c46af55f5dc7e8346a1e39e36a"} Nov 02 17:39:28.912184 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.912092 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87"} Nov 02 17:39:28.925414 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.925305 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-246zz" event={"ID":"68c1bf64-b418-49c2-9753-4fad4bff08ae","Type":"ContainerStarted","Data":"7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e"} Nov 02 17:39:28.936279 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:28.936191 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zb589" event={"ID":"ee56fd64-8479-45f4-b098-16208b7ced9f","Type":"ContainerStarted","Data":"258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef"} Nov 02 17:39:29.132630 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.132497 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:29.276898 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.276682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:29.276898 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.276845 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:29.276898 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.276905 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:31.276887619 +0000 UTC m=+6.064547318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:29.377251 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.377168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:29.377415 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.377277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:29.377482 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.377430 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:29.377482 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.377452 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:29.377581 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.377486 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:29.377581 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.377574 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:31.377542531 +0000 UTC m=+6.165202224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:29.377998 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.377979 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:29.378126 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:29.378031 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:39:31.378015381 +0000 UTC m=+6.165675065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:29.946146 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.946067 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"07e33052aa2d73769dbf3107e42483d7","Type":"ContainerStarted","Data":"36b98675395c780970a62008e4d520290df416f97c74b3e6ec6a6752e3cd5f50"} Nov 02 17:39:29.967031 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.966521 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"3f4115a2f4c3236862e1ddc68e0eda82","Type":"ContainerStarted","Data":"ffdac605b8dc8e0ea5e61d8861ae3a2dbed6feb23eb3c6ab4266f7e640703238"} Nov 02 17:39:29.986483 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:29.986417 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" podStartSLOduration=3.986396708 podStartE2EDuration="3.986396708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:29.964692009 +0000 UTC m=+4.752351714" watchObservedRunningTime="2026-11-02 17:39:29.986396708 +0000 UTC m=+4.774056412" Nov 02 17:39:30.837909 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:30.837879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:30.838064 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:30.837890 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:30.838064 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:30.838011 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:30.838064 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:30.838033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:30.838233 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:30.838140 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:30.846171 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:30.846103 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:30.983597 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:30.981602 2581 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="f333a5e18c68e69b3a3ea36255f6a59afb808533aebd6365524449b8b556884a" exitCode=0 Nov 02 17:39:30.983597 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:30.982835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerDied","Data":"f333a5e18c68e69b3a3ea36255f6a59afb808533aebd6365524449b8b556884a"} Nov 02 17:39:31.009556 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:31.008880 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" podStartSLOduration=5.008861526 podStartE2EDuration="5.008861526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:29.986857294 +0000 UTC m=+4.774517003" watchObservedRunningTime="2026-11-02 17:39:31.008861526 +0000 UTC m=+5.796521232" Nov 02 17:39:31.294297 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:31.294207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:31.294463 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.294404 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:31.294531 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.294469 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:35.294450839 +0000 UTC m=+10.082110522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:31.394953 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:31.394879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:31.394953 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:31.394965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:31.395240 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.395141 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:31.395240 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.395204 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:39:35.395186719 +0000 UTC m=+10.182846402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:31.395695 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.395648 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:31.395695 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.395672 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:31.395695 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.395684 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:31.395951 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:31.395728 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:35.395714517 +0000 UTC m=+10.183374201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:32.837884 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:32.837817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:32.838438 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:32.837954 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:32.838438 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:32.838378 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:32.838550 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:32.838470 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:32.838550 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:32.838522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:32.838659 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:32.838604 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:32.997314 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:32.997275 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkl7h" event={"ID":"20c5900d-985b-4db3-8232-5c24ff85cc01","Type":"ContainerStarted","Data":"e68216e755a458274a267403c26110b005909d388b6eaf95ff45f2ee4b7242ee"} Nov 02 17:39:33.016594 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:33.016533 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rkl7h" podStartSLOduration=4.068227412 podStartE2EDuration="8.016515163s" totalImagesPullingTime="3.948287751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:33.015442022 +0000 UTC m=+7.803101727" watchObservedRunningTime="2026-11-02 17:39:33.016515163 +0000 UTC m=+7.804174870" Nov 02 17:39:34.005202 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:34.005155 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" event={"ID":"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1","Type":"ContainerStarted","Data":"dd287b948b1d316d124d05c9368c5b0ffe202b9cd41b8a539be8acf72d1d29e2"} Nov 02 17:39:34.008795 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:34.008744 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-25nk6" event={"ID":"12700290-2871-451f-9f86-3ea2830de15d","Type":"ContainerStarted","Data":"8571962329a08443d62233b5665c59ae3cc53308e6d225c40d2fe1398b961ed3"} Nov 02 17:39:34.036514 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:34.036455 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-25nk6" podStartSLOduration=3.782191863 podStartE2EDuration="9.036438928s" totalImagesPullingTime="5.254247065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:34.036317096 +0000 UTC m=+8.823976803" watchObservedRunningTime="2026-11-02 17:39:34.036438928 +0000 UTC m=+8.824098632" Nov 02 17:39:34.837332 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:34.837293 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:34.837523 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:34.837442 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:34.838201 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:34.837895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:34.838201 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:34.837996 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:34.838201 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:34.838075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:34.838201 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:34.838147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:35.089520 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.089321 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:39:35.330935 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.330901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:35.331112 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.331097 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:35.331192 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.331155 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:43.331138769 +0000 UTC m=+18.118798456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:35.431882 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.431798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:35.431882 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.431853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:35.432108 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.431977 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:35.432108 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.431982 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:35.432108 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.432011 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:35.432108 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.432024 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:35.432108 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.432042 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:39:43.432024617 +0000 UTC m=+18.219684309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:35.432108 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:35.432096 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:43.432077069 +0000 UTC m=+18.219736774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:35.704460 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.703781 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:39:35.089338212Z","UUID":"83ab0824-0080-4bbb-84ce-48a3edef7c06","Handler":null,"Name":"","Endpoint":""} Nov 02 17:39:35.710110 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.710000 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:39:35.710176 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.710031 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:39:35.976686 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.976487 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:35.977812 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:35.977787 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:36.014721 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.014315 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hqxnx" event={"ID":"8271b561-5608-4f0c-93eb-0850dcdcd9c6","Type":"ContainerStarted","Data":"bbe1ca7330bf9e509963929207fb0ce60cd4ea6e5d3123ae06217df1b01a1b10"} Nov 02 17:39:36.017282 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.017216 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" event={"ID":"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1","Type":"ContainerStarted","Data":"1b35ec549658a6888f7859c54344de25abe10039cb5fd5373f866141cab3799d"} Nov 02 17:39:36.020357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.019809 2581 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="9d313db15c562c99e70863e7a48c02d1c81fa911de931c97013d6f311b29978b" exitCode=0 Nov 02 17:39:36.020357 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.019866 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerDied","Data":"9d313db15c562c99e70863e7a48c02d1c81fa911de931c97013d6f311b29978b"} Nov 02 17:39:36.023160 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.023138 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zb589" event={"ID":"ee56fd64-8479-45f4-b098-16208b7ced9f","Type":"ContainerStarted","Data":"82c55be34518f714c2cebbb1f14e601148838d0e241fc1791c1bf2dc6fa7e677"} Nov 02 17:39:36.034657 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.034603 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hqxnx" podStartSLOduration=3.513291105 podStartE2EDuration="11.034592052s" totalImagesPullingTime="7.521300947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:36.033820691 +0000 UTC m=+10.821480395" watchObservedRunningTime="2026-11-02 17:39:36.034592052 +0000 UTC m=+10.822251754" Nov 02 17:39:36.082685 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.082645 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zb589" podStartSLOduration=3.414713114 podStartE2EDuration="11.082633329s" totalImagesPullingTime="7.667920215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:36.082386853 +0000 UTC m=+10.870046568" watchObservedRunningTime="2026-11-02 17:39:36.082633329 +0000 UTC m=+10.870293031" Nov 02 17:39:36.838787 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.837909 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:36.838787 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:36.838066 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:36.838787 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.838476 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:36.838787 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:36.838578 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:36.838787 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:36.838637 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:36.838787 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:36.838705 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:37.027656 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:37.027628 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" event={"ID":"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1","Type":"ContainerStarted","Data":"b4a052d315cf60e8414fa9c1bc8d9f8a45a399e1292b348d5c074872026d2024"} Nov 02 17:39:37.906769 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:37.906741 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:37.907231 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:37.906873 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:37.907325 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:37.907307 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:39:37.924749 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:37.924711 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" podStartSLOduration=5.057343924 podStartE2EDuration="12.924697636s" totalImagesPullingTime="7.867353712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:37.049456897 +0000 UTC m=+11.837116616" watchObservedRunningTime="2026-11-02 17:39:37.924697636 +0000 UTC m=+12.712357339" Nov 02 17:39:38.047615 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.047576 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-246zz" event={"ID":"68c1bf64-b418-49c2-9753-4fad4bff08ae","Type":"ContainerStarted","Data":"43541e50c07fcbe3b17963f9024fbb4788bb099932171e2989cf017695ea2a3d"} Nov 02 17:39:38.049998 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.049972 2581 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="7ac10d92712a459d316d83f8941fddc826bc7f88b8dd6b085ddccf1f160807ed" exitCode=0 Nov 02 17:39:38.050124 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.050080 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerDied","Data":"7ac10d92712a459d316d83f8941fddc826bc7f88b8dd6b085ddccf1f160807ed"} Nov 02 17:39:38.051992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.051613 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" event={"ID":"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c","Type":"ContainerStarted","Data":"25fa74f7d633489544cb4f5ec749671bd9b6acbb4af1a4ae5a78b6be40d5e0e2"} Nov 02 17:39:38.088791 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.088715 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-246zz" podStartSLOduration=5.201475629 podStartE2EDuration="13.088703508s" totalImagesPullingTime="7.887227879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:38.06720953 +0000 UTC m=+12.854869234" watchObservedRunningTime="2026-11-02 17:39:38.088703508 +0000 UTC m=+12.876363212" Nov 02 17:39:38.089231 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.089201 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" podStartSLOduration=3.266034515 podStartE2EDuration="13.089192769s" totalImagesPullingTime="9.823158254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:38.0883282 +0000 UTC m=+12.875987904" watchObservedRunningTime="2026-11-02 17:39:38.089192769 +0000 UTC m=+12.876852475" Nov 02 17:39:38.837451 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.837420 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:38.837617 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:38.837537 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:38.837947 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.837928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:38.838030 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:38.838010 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:38.838141 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:38.838094 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:38.838225 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:38.838206 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:40.058722 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:40.058684 2581 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="6b0019b580e913210111ee307d3e7996badedeb70ecf81cd05c7b928d5e12160" exitCode=0 Nov 02 17:39:40.059400 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:40.058738 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerDied","Data":"6b0019b580e913210111ee307d3e7996badedeb70ecf81cd05c7b928d5e12160"} Nov 02 17:39:40.838199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:40.837391 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:40.838199 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:40.837784 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:40.838199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:40.837678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:40.838199 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:40.837858 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:40.838199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:40.837669 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:40.838199 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:40.837907 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:42.837716 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:42.837564 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:42.838119 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:42.837606 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:42.838119 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:42.837789 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:42.838119 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:42.837867 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:42.838119 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:42.837612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:42.838119 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:42.837952 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:43.067084 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067039 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:39:43.067411 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067385 2581 generic.go:356] "Generic (PLEG): container finished" podID="98c18ffc-40f4-4360-9095-b08f95cab47c" containerID="230a6af9e10a5bd19f3bc35436a4802c84ca9bfbf16084c97435e49a0cedec83" exitCode=1 Nov 02 17:39:43.067510 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067463 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"03007a9dababcdcdf1aefd6ebc8a4a095bfbf180ab5453ae6e2bcc4df31251fa"} Nov 02 17:39:43.067573 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067530 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"2439b9e306e49646b0ba67b0a0166f07304d6d940a2d88ae03a685a671357976"} Nov 02 17:39:43.067573 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067569 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"0de8fca97268e72eca201862288431732043ac8705b6605448feb232164857dd"} Nov 02 17:39:43.067637 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067607 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"6d66f59622fd7a5b71c69a833ea933d0d81f6838c89effbb6ca018191a3e4222"} Nov 02 17:39:43.067687 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067675 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerDied","Data":"230a6af9e10a5bd19f3bc35436a4802c84ca9bfbf16084c97435e49a0cedec83"} Nov 02 17:39:43.067746 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.067735 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"080bea7ced76bf9da2a132e95452e2de8d87ffcc6981d5f0553318968293ac66"} Nov 02 17:39:43.070291 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.070271 2581 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="e6d0c549701e0a5843b38caa84a257eb981b9cd60695de3baf73c70a08d0a309" exitCode=0 Nov 02 17:39:43.070346 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.070317 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerDied","Data":"e6d0c549701e0a5843b38caa84a257eb981b9cd60695de3baf73c70a08d0a309"} Nov 02 17:39:43.400528 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.400443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:43.400664 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.400591 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:43.400664 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.400650 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:59.400635257 +0000 UTC m=+34.188294938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:43.501698 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.501672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:43.501846 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:43.501723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:43.501846 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.501810 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:43.501846 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.501812 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:43.501846 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.501822 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:43.501846 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.501832 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:43.502022 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.501877 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:39:59.501858521 +0000 UTC m=+34.289518216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:43.502022 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:43.501898 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:59.501887951 +0000 UTC m=+34.289547648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:44.075269 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:44.075232 2581 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="a497b3362b624916aa0ce313175ccab1eed32ea1f96953f10fd39b7c82a2e88c" exitCode=0 Nov 02 17:39:44.075759 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:44.075277 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerDied","Data":"a497b3362b624916aa0ce313175ccab1eed32ea1f96953f10fd39b7c82a2e88c"} Nov 02 17:39:44.837253 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:44.837219 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:44.837253 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:44.837249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:44.837432 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:44.837249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:44.837432 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:44.837323 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:44.837432 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:44.837418 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:44.837537 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:44.837486 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:45.079566 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:45.079493 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:39:45.080036 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:45.079835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"542e576bf368dd5f76545abf8c5acb357f6e999588967896017c3cd5fa89783e"} Nov 02 17:39:45.082897 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:45.082874 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" event={"ID":"094f3261-e65e-4fd0-893d-0e9c0f99988f","Type":"ContainerStarted","Data":"0856d1e767d504e29eb1171f4d761eb2591fa73b0304d4a29006ed0249723442"} Nov 02 17:39:45.112533 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:45.112493 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zw4rm" podStartSLOduration=8.81726785 podStartE2EDuration="20.112481797s" totalImagesPullingTime="11.295213947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:45.110919047 +0000 UTC m=+19.898578763" watchObservedRunningTime="2026-11-02 17:39:45.112481797 +0000 UTC m=+19.900141499" Nov 02 17:39:46.837348 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:46.837173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:46.837859 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:46.837211 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:46.837859 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:46.837427 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:46.837859 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:46.837217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:46.837859 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:46.837492 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:46.837859 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:46.837559 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:47.091085 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:47.090967 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:39:47.091365 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:47.091347 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"78daa1dc32b3b314247d9a1585a7da91a3eaf7ebd4da803cbbe12b89f6118c81"} Nov 02 17:39:47.091630 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:47.091614 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:47.091700 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:47.091689 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:47.091752 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:47.091713 2581 scope.go:122] "RemoveContainer" containerID="230a6af9e10a5bd19f3bc35436a4802c84ca9bfbf16084c97435e49a0cedec83" Nov 02 17:39:47.107400 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:47.107380 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:48.097471 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.097441 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:39:48.097904 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.097801 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" event={"ID":"98c18ffc-40f4-4360-9095-b08f95cab47c","Type":"ContainerStarted","Data":"03aa1a29bfc40a6363729cc592659cc057934bcec12ea2faa03f5eb999009870"} Nov 02 17:39:48.098011 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.097988 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:48.113042 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.113020 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:39:48.132884 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.132846 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" podStartSLOduration=9.014638714 podStartE2EDuration="23.13283489s" totalImagesPullingTime="14.118196176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:48.130983393 +0000 UTC m=+22.918643097" watchObservedRunningTime="2026-11-02 17:39:48.13283489 +0000 UTC m=+22.920494611" Nov 02 17:39:48.522250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.522180 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xmv2b"] Nov 02 17:39:48.522410 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.522286 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:48.522410 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:48.522383 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:48.525627 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.525601 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhqwb"] Nov 02 17:39:48.525736 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.525710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:48.525822 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:48.525803 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:48.526667 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.526649 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89pdp"] Nov 02 17:39:48.526745 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:48.526733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:48.526824 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:48.526806 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:49.837365 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:49.837328 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:49.837861 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:49.837470 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:49.837861 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:49.837514 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:49.837861 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:49.837598 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:50.837289 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:50.837259 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:50.837408 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:50.837369 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:51.837367 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:51.837271 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:51.837519 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:51.837390 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:51.837519 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:51.837441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:51.837946 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:51.837531 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:52.837740 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:52.837665 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:52.838248 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:52.837793 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:53.837849 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:53.837676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:53.838290 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:53.837691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:53.838290 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:53.837929 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:53.838290 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:53.838001 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:54.584644 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:54.584579 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zb589_ee56fd64-8479-45f4-b098-16208b7ced9f/dns-node-resolver/0.log" Nov 02 17:39:54.837686 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:54.837588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:54.837863 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:54.837699 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:55.161532 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:55.161469 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-25nk6_12700290-2871-451f-9f86-3ea2830de15d/node-ca/0.log" Nov 02 17:39:55.838650 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:55.838623 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:55.838837 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:55.838709 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:55.838837 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:55.838752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:55.838837 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:55.838809 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:56.837727 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:56.837691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:56.838131 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:56.837802 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:57.837833 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:57.837802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:57.838251 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:57.837900 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:57.838251 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:57.837989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:57.838251 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:57.838115 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:39:58.837627 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:58.837596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:58.837781 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:58.837720 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:39:59.414124 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:59.414097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:39:59.414499 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.414232 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:59.414499 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.414288 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs podName:9e998028-a165-4d2b-8c14-c422d8a4d099 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:31.414271186 +0000 UTC m=+66.201930881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs") pod "network-metrics-daemon-rhqwb" (UID: "9e998028-a165-4d2b-8c14-c422d8a4d099") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:59.515276 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:59.515251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:59.515405 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:59.515282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:59.515405 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.515379 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:59.515511 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.515414 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:59.515511 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.515441 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:59.515511 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.515455 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8sl6f for pod openshift-network-diagnostics/network-check-target-89pdp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:59.515511 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.515423 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret podName:8bd74c07-03e4-44b3-b5d3-4f06a5acb51c nodeName:}" failed. No retries permitted until 2026-11-02 17:40:31.51540869 +0000 UTC m=+66.303068385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret") pod "global-pull-secret-syncer-xmv2b" (UID: "8bd74c07-03e4-44b3-b5d3-4f06a5acb51c") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:59.515636 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.515516 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f podName:3a654875-bb1c-4b4e-834d-9e8001f2a351 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:31.515501915 +0000 UTC m=+66.303161610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8sl6f" (UniqueName: "kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f") pod "network-check-target-89pdp" (UID: "3a654875-bb1c-4b4e-834d-9e8001f2a351") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:59.837862 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:59.837786 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:39:59.838016 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.837874 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:39:59.838016 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:39:59.837932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:39:59.838158 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:39:59.838022 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:00.837468 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:00.837440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:00.837835 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:00.837538 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:40:01.837361 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:01.837326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:01.837532 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:01.837459 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:01.837789 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:01.837526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:01.837789 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:01.837613 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:40:02.837892 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:02.837861 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:02.838300 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:02.837969 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:40:03.838035 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:03.838005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:03.838035 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:03.838040 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:03.838456 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:03.838138 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:40:03.838456 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:03.838275 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:04.837474 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:04.837447 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:04.837644 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:04.837548 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:40:05.838323 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:05.838286 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:05.838686 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:05.838423 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:05.838686 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:05.838478 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:05.838686 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:05.838576 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:40:06.837380 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:06.837352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:06.837506 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:06.837450 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:40:07.837453 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:07.837424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:07.837856 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:07.837424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:07.837856 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:07.837517 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:40:07.837856 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:07.837590 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:08.837789 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:08.837759 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:08.838323 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:08.837881 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:40:09.837277 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:09.837245 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:09.837459 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:09.837339 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:40:09.837459 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:09.837423 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:09.837592 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:09.837527 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:10.837141 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:10.837109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:10.837573 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:10.837232 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhqwb" podUID="9e998028-a165-4d2b-8c14-c422d8a4d099" Nov 02 17:40:11.837781 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:11.837749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:11.837781 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:11.837788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:11.838289 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:11.837890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmv2b" podUID="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" Nov 02 17:40:11.838289 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:11.838005 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89pdp" podUID="3a654875-bb1c-4b4e-834d-9e8001f2a351" Nov 02 17:40:12.046372 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.046352 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeReady" Nov 02 17:40:12.046493 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.046461 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:40:12.105856 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.105799 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-88nck"] Nov 02 17:40:12.107252 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.107237 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.118669 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.118652 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:40:12.118758 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.118692 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gjv4t\"" Nov 02 17:40:12.120080 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.120061 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kk5x9"] Nov 02 17:40:12.121236 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.121221 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:40:12.121402 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.121388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.121474 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.121452 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:40:12.125708 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.125687 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88nck"] Nov 02 17:40:12.141251 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.141219 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:40:12.141576 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.141551 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:40:12.141801 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.141787 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:40:12.142707 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.142663 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kk5x9"] Nov 02 17:40:12.143541 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.143522 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mgw6g"] Nov 02 17:40:12.145296 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.145281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.162444 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.162422 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgw6g"] Nov 02 17:40:12.206758 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.206736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40498a67-6c10-479b-b130-3c97cdd76774-cert\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.206864 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.206801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnjw\" (UniqueName: \"kubernetes.io/projected/40498a67-6c10-479b-b130-3c97cdd76774-kube-api-access-xvnjw\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.218580 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.218557 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:40:12.218662 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.218651 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs2ss\"" Nov 02 17:40:12.219704 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.219689 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7flsb\"" Nov 02 17:40:12.219751 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.219699 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:40:12.221820 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.221806 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:40:12.308043 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72cc37d8-2509-4902-8494-415871c9db22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.308137 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72cc37d8-2509-4902-8494-415871c9db22-crio-socket\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.308137 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-tmp-dir\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.308205 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72cc37d8-2509-4902-8494-415871c9db22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.308205 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40498a67-6c10-479b-b130-3c97cdd76774-cert\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.308205 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308195 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-metrics-tls\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.308310 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6q6\" (UniqueName: \"kubernetes.io/projected/72cc37d8-2509-4902-8494-415871c9db22-kube-api-access-kn6q6\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.308310 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72cc37d8-2509-4902-8494-415871c9db22-data-volume\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.308402 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnjw\" (UniqueName: \"kubernetes.io/projected/40498a67-6c10-479b-b130-3c97cdd76774-kube-api-access-xvnjw\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.308434 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-config-volume\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.308434 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.308427 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4fr\" (UniqueName: \"kubernetes.io/projected/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-kube-api-access-jj4fr\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.311988 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.311969 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40498a67-6c10-479b-b130-3c97cdd76774-cert\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.316879 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.316859 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnjw\" (UniqueName: \"kubernetes.io/projected/40498a67-6c10-479b-b130-3c97cdd76774-kube-api-access-xvnjw\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.408958 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.408912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-metrics-tls\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.408958 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.408941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6q6\" (UniqueName: \"kubernetes.io/projected/72cc37d8-2509-4902-8494-415871c9db22-kube-api-access-kn6q6\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.408958 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.408960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72cc37d8-2509-4902-8494-415871c9db22-data-volume\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.409178 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409011 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-config-volume\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.409178 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4fr\" (UniqueName: \"kubernetes.io/projected/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-kube-api-access-jj4fr\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.409178 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72cc37d8-2509-4902-8494-415871c9db22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.409178 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72cc37d8-2509-4902-8494-415871c9db22-crio-socket\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.409178 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-tmp-dir\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.409414 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72cc37d8-2509-4902-8494-415871c9db22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.409414 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72cc37d8-2509-4902-8494-415871c9db22-crio-socket\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.409414 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409326 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72cc37d8-2509-4902-8494-415871c9db22-data-volume\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.409580 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409561 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-tmp-dir\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.409717 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409692 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-config-volume\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.409817 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.409692 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72cc37d8-2509-4902-8494-415871c9db22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.411320 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.411296 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72cc37d8-2509-4902-8494-415871c9db22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.411490 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.411475 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-metrics-tls\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.416293 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.416266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:12.416908 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.416887 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6q6\" (UniqueName: \"kubernetes.io/projected/72cc37d8-2509-4902-8494-415871c9db22-kube-api-access-kn6q6\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.417239 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.417219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4fr\" (UniqueName: \"kubernetes.io/projected/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-kube-api-access-jj4fr\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.432459 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.432434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:12.456070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.455922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:12.701870 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.701805 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88nck"] Nov 02 17:40:12.703745 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.703724 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kk5x9"] Nov 02 17:40:12.705541 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:12.705523 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40498a67_6c10_479b_b130_3c97cdd76774.slice/crio-9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309 WatchSource:0}: Error finding container 9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309: Status 404 returned error can't find the container with id 9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309 Nov 02 17:40:12.706667 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:12.706584 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cc37d8_2509_4902_8494_415871c9db22.slice/crio-d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c WatchSource:0}: Error finding container d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c: Status 404 returned error can't find the container with id d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c Nov 02 17:40:12.747322 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.747300 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgw6g"] Nov 02 17:40:12.750084 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:12.750058 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6440b09a_ee4a_4ffa_8102_da0d1d3e10bb.slice/crio-d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9 WatchSource:0}: Error finding container d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9: Status 404 returned error can't find the container with id d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9 Nov 02 17:40:12.837933 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.837903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:12.850282 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.850263 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:40:12.850373 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.850346 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mg2x\"" Nov 02 17:40:12.966629 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.966570 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5f44b9cf-n4s6q"] Nov 02 17:40:12.968017 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.967991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:12.972226 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972204 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 17:40:12.972334 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972237 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 17:40:12.972397 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972245 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ntx78\"" Nov 02 17:40:12.972397 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972247 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 17:40:12.972397 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972291 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 17:40:12.972397 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972326 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:40:12.972397 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.972330 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:40:12.978257 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.978231 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 17:40:12.978684 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.978669 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 17:40:12.986023 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:12.986002 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5f44b9cf-n4s6q"] Nov 02 17:40:13.114406 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114406 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114409 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114616 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114616 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114616 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114706 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114706 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.114706 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.114697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.147107 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.147079 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88nck" event={"ID":"40498a67-6c10-479b-b130-3c97cdd76774","Type":"ContainerStarted","Data":"9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309"} Nov 02 17:40:13.148343 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.148312 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgw6g" event={"ID":"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb","Type":"ContainerStarted","Data":"d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9"} Nov 02 17:40:13.149803 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.149783 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kk5x9" event={"ID":"72cc37d8-2509-4902-8494-415871c9db22","Type":"ContainerStarted","Data":"6f7d9237319daa41b1b08b5865399dbb9d5d4d50665e09fc65c410a01d374374"} Nov 02 17:40:13.149887 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.149825 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kk5x9" event={"ID":"72cc37d8-2509-4902-8494-415871c9db22","Type":"ContainerStarted","Data":"d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c"} Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.215881 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.215768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.216542 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.216314 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.216542 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.216342 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.216706 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.216636 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.216932 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.216911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.217080 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.217036 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.218575 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.218552 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.219166 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.219146 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.226024 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.226002 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.277080 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.277038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:13.562727 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.562696 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5f44b9cf-n4s6q"] Nov 02 17:40:13.565933 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:13.565880 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a34000_505d_44f9_95fc_2a1f3cb4e813.slice/crio-56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9 WatchSource:0}: Error finding container 56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9: Status 404 returned error can't find the container with id 56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9 Nov 02 17:40:13.837450 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.837415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:13.838095 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.838070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:13.842356 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.842333 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:40:13.842511 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.842492 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:40:13.843869 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.843847 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:40:13.843963 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.843936 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mkftb\"" Nov 02 17:40:13.943199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:13.943143 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6x7d8\"" Nov 02 17:40:14.156517 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:14.156381 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kk5x9" event={"ID":"72cc37d8-2509-4902-8494-415871c9db22","Type":"ContainerStarted","Data":"d9e9257ac5882ae59773d4409b64e07d382fc5878054ec2b2716e5a9d8488a9d"} Nov 02 17:40:14.159769 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:14.159708 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5f44b9cf-n4s6q" event={"ID":"67a34000-505d-44f9-95fc-2a1f3cb4e813","Type":"ContainerStarted","Data":"56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9"} Nov 02 17:40:15.165324 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:15.164620 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgw6g" event={"ID":"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb","Type":"ContainerStarted","Data":"7b16a47d2bcb0b567f86a410251e044ad0ad2c8bee90ba27c5dc81083ac82853"} Nov 02 17:40:15.165324 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:15.164685 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgw6g" event={"ID":"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb","Type":"ContainerStarted","Data":"31854f9a4616180571454849bb882009838d55b7efaa200feb3bd7f49e756d2a"} Nov 02 17:40:15.165324 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:15.165134 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:15.186342 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:15.186295 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-mgw6g" podStartSLOduration=1.152487538 podStartE2EDuration="3.186283024s" totalImagesPullingTime="2.033795486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:15.185260138 +0000 UTC m=+49.972919842" watchObservedRunningTime="2026-11-02 17:40:15.186283024 +0000 UTC m=+49.973942727" Nov 02 17:40:16.170577 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:16.170539 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kk5x9" event={"ID":"72cc37d8-2509-4902-8494-415871c9db22","Type":"ContainerStarted","Data":"31928263c15e0138539966aa8524ebe98d803a5a5a61da30f262530a282fb850"} Nov 02 17:40:16.172590 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:16.172559 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88nck" event={"ID":"40498a67-6c10-479b-b130-3c97cdd76774","Type":"ContainerStarted","Data":"99984225ee9c35fce0fedcc32b168e2a11e8a0d89b05ed58c8494f4403119907"} Nov 02 17:40:16.191911 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:16.191848 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kk5x9" podStartSLOduration=1.799932655 podStartE2EDuration="4.19183597s" totalImagesPullingTime="2.391903315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:16.191526511 +0000 UTC m=+50.979186215" watchObservedRunningTime="2026-11-02 17:40:16.19183597 +0000 UTC m=+50.979495674" Nov 02 17:40:16.210330 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:16.210286 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-88nck" podStartSLOduration=1.197316429 podStartE2EDuration="4.210274523s" totalImagesPullingTime="3.012958094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:16.209928348 +0000 UTC m=+50.997588051" watchObservedRunningTime="2026-11-02 17:40:16.210274523 +0000 UTC m=+50.997934223" Nov 02 17:40:17.178014 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:17.177977 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5f44b9cf-n4s6q" event={"ID":"67a34000-505d-44f9-95fc-2a1f3cb4e813","Type":"ContainerStarted","Data":"d67ea39738ef526880089f146d59feb7687cda2f83f2e84190841a04219ac4c6"} Nov 02 17:40:17.205161 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:17.205116 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b5f44b9cf-n4s6q" podStartSLOduration=2.094718578 podStartE2EDuration="5.205103565s" totalImagesPullingTime="3.110384987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:17.203027786 +0000 UTC m=+51.990687489" watchObservedRunningTime="2026-11-02 17:40:17.205103565 +0000 UTC m=+51.992763267" Nov 02 17:40:17.658508 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:17.658480 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b5f44b9cf-n4s6q_67a34000-505d-44f9-95fc-2a1f3cb4e813/console/0.log" Nov 02 17:40:18.260311 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:18.260246 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgw6g_6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/dns/0.log" Nov 02 17:40:18.459405 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:18.459373 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgw6g_6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/kube-rbac-proxy/0.log" Nov 02 17:40:19.856879 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:19.856854 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zb589_ee56fd64-8479-45f4-b098-16208b7ced9f/dns-node-resolver/0.log" Nov 02 17:40:20.112455 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:20.112391 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:20.656887 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:20.656861 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-25nk6_12700290-2871-451f-9f86-3ea2830de15d/node-ca/0.log" Nov 02 17:40:21.656732 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:21.656694 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88nck_40498a67-6c10-479b-b130-3c97cdd76774/serve-healthcheck-canary/0.log" Nov 02 17:40:22.673233 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.673199 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6"] Nov 02 17:40:22.676441 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.676424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.680978 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.680935 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:40:22.682519 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.682502 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:40:22.682627 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.682521 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:40:22.682682 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.682528 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:40:22.682682 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.682534 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wbwjw\"" Nov 02 17:40:22.682682 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.682588 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:40:22.685716 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.685695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pskb\" (UniqueName: \"kubernetes.io/projected/31b1e410-3fff-449a-a344-8ca439c0c9af-kube-api-access-4pskb\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.685811 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.685725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.685811 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.685799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b1e410-3fff-449a-a344-8ca439c0c9af-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.685885 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.685827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.689259 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.689239 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6"] Nov 02 17:40:22.786582 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.786559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b1e410-3fff-449a-a344-8ca439c0c9af-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.786700 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.786588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.786700 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.786626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pskb\" (UniqueName: \"kubernetes.io/projected/31b1e410-3fff-449a-a344-8ca439c0c9af-kube-api-access-4pskb\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.786700 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.786648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.786811 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:22.786699 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Nov 02 17:40:22.786811 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:22.786804 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls podName:31b1e410-3fff-449a-a344-8ca439c0c9af nodeName:}" failed. No retries permitted until 2026-11-02 17:40:23.286785857 +0000 UTC m=+58.074445556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-n2gp6" (UID: "31b1e410-3fff-449a-a344-8ca439c0c9af") : secret "prometheus-operator-tls" not found Nov 02 17:40:22.787257 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.787233 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b1e410-3fff-449a-a344-8ca439c0c9af-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.788893 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.788866 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:22.798843 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:22.798817 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pskb\" (UniqueName: \"kubernetes.io/projected/31b1e410-3fff-449a-a344-8ca439c0c9af-kube-api-access-4pskb\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:23.278010 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.277977 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:23.278181 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.278027 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:23.282513 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.282491 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:23.289636 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.289609 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:23.291961 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.291940 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:23.590798 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.590769 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:23.858917 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:23.858839 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6"] Nov 02 17:40:23.862570 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:23.862544 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b1e410_3fff_449a_a344_8ca439c0c9af.slice/crio-9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba WatchSource:0}: Error finding container 9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba: Status 404 returned error can't find the container with id 9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba Nov 02 17:40:24.199692 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:24.199611 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" event={"ID":"31b1e410-3fff-449a-a344-8ca439c0c9af","Type":"ContainerStarted","Data":"9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba"} Nov 02 17:40:24.203486 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:24.203461 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:25.205020 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:25.204988 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" event={"ID":"31b1e410-3fff-449a-a344-8ca439c0c9af","Type":"ContainerStarted","Data":"a0d368c85cbff12c6314f027379e17fb1c88fbcacea131426fa49806e318a240"} Nov 02 17:40:25.205365 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:25.205061 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" event={"ID":"31b1e410-3fff-449a-a344-8ca439c0c9af","Type":"ContainerStarted","Data":"9e61d6552c26343af1eca6d23c1f905cb889a8b8817fbcf30ea07d1a27fcc234"} Nov 02 17:40:25.223551 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:25.223511 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" podStartSLOduration=2.03871181 podStartE2EDuration="3.223498877s" totalImagesPullingTime="1.184787067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:25.221922189 +0000 UTC m=+60.009581892" watchObservedRunningTime="2026-11-02 17:40:25.223498877 +0000 UTC m=+60.011158580" Nov 02 17:40:26.180706 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:26.180679 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:27.038100 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.038073 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5"] Nov 02 17:40:27.042211 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.042190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.047156 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.047138 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 17:40:27.050135 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.050115 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qqmv6\"" Nov 02 17:40:27.051000 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.050978 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd"] Nov 02 17:40:27.053813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.053789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.053921 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.053813 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:40:27.056331 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.056310 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7jkqt"] Nov 02 17:40:27.059326 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.059309 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5"] Nov 02 17:40:27.059424 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.059413 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.063450 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.063431 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:40:27.066944 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.066912 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:40:27.067070 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.067037 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:40:27.067166 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.067149 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5qkwb\"" Nov 02 17:40:27.068313 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.068293 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd"] Nov 02 17:40:27.088837 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.088810 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-84l5b\"" Nov 02 17:40:27.089001 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.088982 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:40:27.089101 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.089077 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:40:27.119032 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.119199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxvz\" (UniqueName: \"kubernetes.io/projected/a5a5d720-67fb-459f-823f-ad9993dec96c-kube-api-access-2fxvz\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.119199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.119199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-textfile\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-metrics-client-ca\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119199 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.119352 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-sys\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119352 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.119410 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.119410 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.119472 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119409 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119533 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-root\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119566 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-tls\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119597 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5d720-67fb-459f-823f-ad9993dec96c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.119628 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119617 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqvg\" (UniqueName: \"kubernetes.io/projected/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-kube-api-access-sbqvg\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119661 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.119692 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119656 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-wtmp\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.119724 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbdb\" (UniqueName: \"kubernetes.io/projected/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-api-access-lpbdb\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.119762 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.119748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.165903 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.165868 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:40:27.220100 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxvz\" (UniqueName: \"kubernetes.io/projected/a5a5d720-67fb-459f-823f-ad9993dec96c-kube-api-access-2fxvz\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.220250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-textfile\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-metrics-client-ca\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.220250 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-sys\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220524 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220524 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220625 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220625 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220625 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220610 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-textfile\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220773 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-root\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220773 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-root\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220773 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-tls\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220773 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5d720-67fb-459f-823f-ad9993dec96c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.220773 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220752 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqvg\" (UniqueName: \"kubernetes.io/projected/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-kube-api-access-sbqvg\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.220992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-wtmp\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220858 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbdb\" (UniqueName: \"kubernetes.io/projected/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-api-access-lpbdb\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.220992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220920 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.220992 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.220960 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-metrics-client-ca\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.221295 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.221233 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-sys\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.221349 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.221293 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.222327 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.221781 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-wtmp\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.222327 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.222070 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.222327 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.222145 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5d720-67fb-459f-823f-ad9993dec96c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.222327 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.222278 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.223087 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.223061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.223469 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.223446 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-tls\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.223605 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.223582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.224184 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.224164 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.224271 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.224216 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.224547 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.224532 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.233417 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.233394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqvg\" (UniqueName: \"kubernetes.io/projected/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-kube-api-access-sbqvg\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.233621 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.233601 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbdb\" (UniqueName: \"kubernetes.io/projected/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-api-access-lpbdb\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.234012 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.233996 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxvz\" (UniqueName: \"kubernetes.io/projected/a5a5d720-67fb-459f-823f-ad9993dec96c-kube-api-access-2fxvz\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.357436 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.357400 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:27.380868 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.380839 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:27.387086 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.387009 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:27.396870 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:27.396845 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec52e3c_2a94_48a7_b568_ab54be3ccb9e.slice/crio-5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e WatchSource:0}: Error finding container 5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e: Status 404 returned error can't find the container with id 5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e Nov 02 17:40:27.636493 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.636407 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5"] Nov 02 17:40:27.640015 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:27.639990 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a5d720_67fb_459f_823f_ad9993dec96c.slice/crio-a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848 WatchSource:0}: Error finding container a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848: Status 404 returned error can't find the container with id a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848 Nov 02 17:40:27.659214 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:27.659190 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd"] Nov 02 17:40:27.669001 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:27.668972 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5701f7c4_97ba_4887_b9fe_4e3bc9941acb.slice/crio-97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316 WatchSource:0}: Error finding container 97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316: Status 404 returned error can't find the container with id 97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316 Nov 02 17:40:28.137138 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.137101 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:40:28.141260 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.141238 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.148958 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.148918 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:40:28.160079 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.157732 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:40:28.182725 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.182699 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:40:28.182874 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.182767 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:40:28.182933 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.182891 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:40:28.182982 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.182931 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:40:28.183034 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.183008 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:40:28.183108 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.183095 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:40:28.183158 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.183096 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:40:28.183158 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.183137 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:40:28.183254 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.183206 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zmf58\"" Nov 02 17:40:28.221822 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.221761 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jkqt" event={"ID":"aec52e3c-2a94-48a7-b568-ab54be3ccb9e","Type":"ContainerStarted","Data":"5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e"} Nov 02 17:40:28.223728 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.223653 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" event={"ID":"5701f7c4-97ba-4887-b9fe-4e3bc9941acb","Type":"ContainerStarted","Data":"97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316"} Nov 02 17:40:28.228525 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.228397 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" event={"ID":"a5a5d720-67fb-459f-823f-ad9993dec96c","Type":"ContainerStarted","Data":"a7e49a4039e001aa3c337340e18fc85444832d44d328e0ef521582452f68bf1f"} Nov 02 17:40:28.228525 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.228464 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" event={"ID":"a5a5d720-67fb-459f-823f-ad9993dec96c","Type":"ContainerStarted","Data":"60c8ec0ebf3fe4106d16ffa00be16e6fa973b46e899e12b299bbba11271eb6e0"} Nov 02 17:40:28.228525 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.228493 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" event={"ID":"a5a5d720-67fb-459f-823f-ad9993dec96c","Type":"ContainerStarted","Data":"a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848"} Nov 02 17:40:28.229829 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.229798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.229955 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.229930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230023 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.229999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230127 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230187 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230243 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230243 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230352 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230352 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230352 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.230500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.230438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.331848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.331914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.331960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.331983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332149 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.332458 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.332299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.333221 ip-10-0-141-84 kubenswrapper[2581]: E1102 17:40:28.332970 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle podName:f8aaaba8-6e2f-4193-ac60-8f930a419e00 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:28.832946913 +0000 UTC m=+63.620606620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00") : configmap references non-existent config key: ca-bundle.crt Nov 02 17:40:28.334034 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.334002 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.335339 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.334285 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.335339 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.335295 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.335506 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.335357 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.336609 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.336566 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.336930 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.336906 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.337005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.336943 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.337116 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.337093 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.337523 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.337502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.338191 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.338162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.338284 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.338232 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.354994 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.354939 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.837302 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.837256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:28.838136 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:28.838107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:29.071252 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.071218 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:29.238905 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.238871 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" event={"ID":"5701f7c4-97ba-4887-b9fe-4e3bc9941acb","Type":"ContainerStarted","Data":"bec1e9957a24b579a46be965c9ec86471b59a2be536e9c748b98a8a1a2ca5aff"} Nov 02 17:40:29.239377 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.238967 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" event={"ID":"5701f7c4-97ba-4887-b9fe-4e3bc9941acb","Type":"ContainerStarted","Data":"9c5b75301590d6fe9965f58df46b318222b1cdfda11eaca59d667837c7c331e6"} Nov 02 17:40:29.241861 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.241835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" event={"ID":"a5a5d720-67fb-459f-823f-ad9993dec96c","Type":"ContainerStarted","Data":"e9a73374a4caa6c38e4c93c7dfa7500a0ab17833e447eedf8a87cb9d189865f4"} Nov 02 17:40:29.248074 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.247585 2581 generic.go:356] "Generic (PLEG): container finished" podID="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" containerID="ab37125646669f2fc7bcd4aac79a7f2e06ae5c6bce637e5e1d0e68ff65a1cd98" exitCode=0 Nov 02 17:40:29.248074 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.247626 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jkqt" event={"ID":"aec52e3c-2a94-48a7-b568-ab54be3ccb9e","Type":"ContainerDied","Data":"ab37125646669f2fc7bcd4aac79a7f2e06ae5c6bce637e5e1d0e68ff65a1cd98"} Nov 02 17:40:29.270789 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.270733 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" podStartSLOduration=1.155126728 podStartE2EDuration="2.270713373s" totalImagesPullingTime="1.115586645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:29.269538007 +0000 UTC m=+64.057197712" watchObservedRunningTime="2026-11-02 17:40:29.270713373 +0000 UTC m=+64.058373077" Nov 02 17:40:29.365042 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:29.364868 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:40:29.367725 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:29.367699 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aaaba8_6e2f_4193_ac60_8f930a419e00.slice/crio-85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e WatchSource:0}: Error finding container 85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e: Status 404 returned error can't find the container with id 85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e Nov 02 17:40:30.253846 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:30.253810 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jkqt" event={"ID":"aec52e3c-2a94-48a7-b568-ab54be3ccb9e","Type":"ContainerStarted","Data":"9ad7e4ae74f55ea69c83209ef3601b2f40de9c3d3b8d6d583d85ebb5d0772d06"} Nov 02 17:40:30.254312 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:30.253879 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7jkqt" event={"ID":"aec52e3c-2a94-48a7-b568-ab54be3ccb9e","Type":"ContainerStarted","Data":"b208d113ff3c2dad38acb788d11ec91dbb733479dd0867b091074bb878d83bee"} Nov 02 17:40:30.256286 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:30.256259 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" event={"ID":"5701f7c4-97ba-4887-b9fe-4e3bc9941acb","Type":"ContainerStarted","Data":"16deb7888409d648245b1481be7f04821690d35d843558e6059b0eaaf2649289"} Nov 02 17:40:30.257704 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:30.257674 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e"} Nov 02 17:40:30.276922 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:30.276876 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7jkqt" podStartSLOduration=2.180702509 podStartE2EDuration="3.276863957s" totalImagesPullingTime="1.019798116s" totalInitContainerRuntime="76.363332ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:30.275936832 +0000 UTC m=+65.063596556" watchObservedRunningTime="2026-11-02 17:40:30.276863957 +0000 UTC m=+65.064523707" Nov 02 17:40:30.298677 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:30.298640 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" podStartSLOduration=1.8564647079999999 podStartE2EDuration="3.298628015s" totalImagesPullingTime="1.442163307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:30.297719893 +0000 UTC m=+65.085379608" watchObservedRunningTime="2026-11-02 17:40:30.298628015 +0000 UTC m=+65.086287717" Nov 02 17:40:31.266708 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.266668 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed" exitCode=0 Nov 02 17:40:31.267168 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.266758 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed"} Nov 02 17:40:31.459220 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.459182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:31.462005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.461983 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:40:31.471707 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.471684 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:31.560312 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.560236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:31.560312 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.560272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:31.563112 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.563092 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:40:31.563192 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.563154 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:40:31.573501 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.573473 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:31.573874 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.573857 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:40:31.584079 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.584040 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:31.750854 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.750823 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mg2x\"" Nov 02 17:40:31.757940 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.757911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:31.849602 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.849571 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dls4w"] Nov 02 17:40:31.853155 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.853134 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:31.854013 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.853991 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mkftb\"" Nov 02 17:40:31.861594 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.861572 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:31.861866 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.861853 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6x7d8\"" Nov 02 17:40:31.864223 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.864187 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dls4w"] Nov 02 17:40:31.867686 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.867585 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zwhs4\"" Nov 02 17:40:31.867686 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.867661 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:40:31.869568 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.869537 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:31.963754 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.963718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd8dc667-bb79-4137-845c-4cff317422ed-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:31.963876 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:31.963772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbml9\" (UniqueName: \"kubernetes.io/projected/bd8dc667-bb79-4137-845c-4cff317422ed-kube-api-access-mbml9\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:32.032383 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.032219 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhqwb"] Nov 02 17:40:32.034686 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:32.034660 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e998028_a165_4d2b_8c14_c422d8a4d099.slice/crio-2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e WatchSource:0}: Error finding container 2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e: Status 404 returned error can't find the container with id 2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e Nov 02 17:40:32.064988 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.064922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd8dc667-bb79-4137-845c-4cff317422ed-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:32.064988 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.064965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbml9\" (UniqueName: \"kubernetes.io/projected/bd8dc667-bb79-4137-845c-4cff317422ed-kube-api-access-mbml9\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:32.067649 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.067629 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd8dc667-bb79-4137-845c-4cff317422ed-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:32.075264 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.075241 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbml9\" (UniqueName: \"kubernetes.io/projected/bd8dc667-bb79-4137-845c-4cff317422ed-kube-api-access-mbml9\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:32.142929 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.142897 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xmv2b"] Nov 02 17:40:32.148808 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:32.148782 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd74c07_03e4_44b3_b5d3_4f06a5acb51c.slice/crio-09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827 WatchSource:0}: Error finding container 09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827: Status 404 returned error can't find the container with id 09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827 Nov 02 17:40:32.165452 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.165425 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:32.167825 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.167804 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89pdp"] Nov 02 17:40:32.170809 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:32.170786 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a654875_bb1c_4b4e_834d_9e8001f2a351.slice/crio-2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89 WatchSource:0}: Error finding container 2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89: Status 404 returned error can't find the container with id 2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89 Nov 02 17:40:32.273453 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.273408 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhqwb" event={"ID":"9e998028-a165-4d2b-8c14-c422d8a4d099","Type":"ContainerStarted","Data":"2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e"} Nov 02 17:40:32.275471 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.275438 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89pdp" event={"ID":"3a654875-bb1c-4b4e-834d-9e8001f2a351","Type":"ContainerStarted","Data":"2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89"} Nov 02 17:40:32.277133 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.277102 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xmv2b" event={"ID":"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c","Type":"ContainerStarted","Data":"09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827"} Nov 02 17:40:32.286007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.285978 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-ffdb45754-7bx92"] Nov 02 17:40:32.291011 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.290990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.298078 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.297696 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 17:40:32.303549 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.303529 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 17:40:32.307509 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.307484 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ffdb45754-7bx92"] Nov 02 17:40:32.367242 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-federate-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367398 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367398 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367340 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367398 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367523 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367557 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367524 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45684\" (UniqueName: \"kubernetes.io/projected/4a7e0208-a8bc-43f2-8921-5f3d0909120a-kube-api-access-45684\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367598 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-metrics-client-ca\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.367628 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.367603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-serving-certs-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.389564 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.389434 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 17:40:32.389564 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.389490 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 17:40:32.389564 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.389501 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 17:40:32.389564 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.389518 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 17:40:32.389564 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.389523 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lg2nk\"" Nov 02 17:40:32.443353 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.443326 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dls4w"] Nov 02 17:40:32.446559 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:32.446531 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8dc667_bb79_4137_845c_4cff317422ed.slice/crio-1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd WatchSource:0}: Error finding container 1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd: Status 404 returned error can't find the container with id 1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd Nov 02 17:40:32.468519 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45684\" (UniqueName: \"kubernetes.io/projected/4a7e0208-a8bc-43f2-8921-5f3d0909120a-kube-api-access-45684\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468626 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468530 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-metrics-client-ca\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468626 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-serving-certs-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468626 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-federate-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468758 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468758 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468758 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.468758 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.468732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.470522 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.470428 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-metrics-client-ca\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.470635 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.470593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.474068 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.471271 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-serving-certs-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.477547 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.477019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.477547 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.477113 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.477547 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.477124 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-federate-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.478692 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.478659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.479991 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.479973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45684\" (UniqueName: \"kubernetes.io/projected/4a7e0208-a8bc-43f2-8921-5f3d0909120a-kube-api-access-45684\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.610280 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.610244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:32.914900 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:32.914698 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ffdb45754-7bx92"] Nov 02 17:40:32.928643 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:32.928603 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7e0208_a8bc_43f2_8921_5f3d0909120a.slice/crio-f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f WatchSource:0}: Error finding container f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f: Status 404 returned error can't find the container with id f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f Nov 02 17:40:33.288813 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.288780 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" event={"ID":"4a7e0208-a8bc-43f2-8921-5f3d0909120a","Type":"ContainerStarted","Data":"f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f"} Nov 02 17:40:33.292767 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.292728 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" event={"ID":"bd8dc667-bb79-4137-845c-4cff317422ed","Type":"ContainerStarted","Data":"1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd"} Nov 02 17:40:33.399460 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.399428 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:40:33.407931 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.407909 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.416828 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.416659 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:40:33.416987 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.416968 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:40:33.417143 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.417118 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:40:33.417276 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.417255 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:40:33.418372 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.418115 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:40:33.418372 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.418278 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:40:33.418372 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.418342 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:40:33.421611 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.421102 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:40:33.421611 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.421155 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:40:33.421611 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.421565 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5pnjw\"" Nov 02 17:40:33.421611 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.421573 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:40:33.423761 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.423697 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:40:33.423761 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.423698 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:40:33.425696 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.425489 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bb3vd27e74ksj\"" Nov 02 17:40:33.425696 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.425557 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:40:33.427749 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.426917 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:40:33.427749 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.427113 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:40:33.429176 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.429155 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482936 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.487500 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.482998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.488369 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.483058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.488369 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.483088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.488369 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.483112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.488369 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.483136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.583915 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.583887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584030 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.583959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584030 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.583991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584030 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584017 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584230 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584230 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584230 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584230 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584230 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584336 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584470 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584882 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584882 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.584882 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.584538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.589998 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.586244 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.589998 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.589383 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.592779 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.590763 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.592779 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.591492 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.592779 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.592219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.592779 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.592779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.594307 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.593505 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.594307 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.593956 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.596106 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.594903 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.597138 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.596396 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.597138 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.597088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.598273 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.597717 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.599666 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.599632 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.600695 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.600597 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.600695 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.600640 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.607020 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.606964 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.607139 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.607090 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.608032 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.607580 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.608542 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.608517 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.610016 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.609119 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:33.758742 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:33.758710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:34.081557 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.081392 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:40:34.085926 ip-10-0-141-84 kubenswrapper[2581]: W1102 17:40:34.085895 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bcced4_e7fa_44db_ba68_001da2189ec6.slice/crio-51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0 WatchSource:0}: Error finding container 51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0: Status 404 returned error can't find the container with id 51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0 Nov 02 17:40:34.303505 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.303428 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e"} Nov 02 17:40:34.303505 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.303502 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821"} Nov 02 17:40:34.303982 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.303546 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba"} Nov 02 17:40:34.303982 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.303586 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888"} Nov 02 17:40:34.303982 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.303619 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51"} Nov 02 17:40:34.306160 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.306128 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhqwb" event={"ID":"9e998028-a165-4d2b-8c14-c422d8a4d099","Type":"ContainerStarted","Data":"eb64d36df1ddb12056d19e6a229403941deb5e6fa1f7f80d5e067067b4fe1634"} Nov 02 17:40:34.306278 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.306179 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhqwb" event={"ID":"9e998028-a165-4d2b-8c14-c422d8a4d099","Type":"ContainerStarted","Data":"100152a2c5cc7b5ad5721828a83b925313c175cbebb9518612aabd329e948c35"} Nov 02 17:40:34.313302 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.312560 2581 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="fb843d2f3022fb6c02b264443328dc82aa8488bf1ba72d318eb26ed758c057b4" exitCode=0 Nov 02 17:40:34.313302 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.312601 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"fb843d2f3022fb6c02b264443328dc82aa8488bf1ba72d318eb26ed758c057b4"} Nov 02 17:40:34.313302 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.312664 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0"} Nov 02 17:40:34.324588 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:34.324535 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rhqwb" podStartSLOduration=67.606665519 podStartE2EDuration="1m9.324521979s" totalImagesPullingTime="1.71785646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:34.324001667 +0000 UTC m=+69.111661369" watchObservedRunningTime="2026-11-02 17:40:34.324521979 +0000 UTC m=+69.112181682" Nov 02 17:40:35.325005 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:35.324972 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" event={"ID":"bd8dc667-bb79-4137-845c-4cff317422ed","Type":"ContainerStarted","Data":"7058ebb9ef4379bc7b34db1a47a3e61615ef989284a8e272b74d09ea9bf86548"} Nov 02 17:40:35.326146 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:35.326075 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:35.333346 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:35.333308 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:35.339103 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:35.339078 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerStarted","Data":"5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5"} Nov 02 17:40:35.346652 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:35.346599 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" podStartSLOduration=1.636491468 podStartE2EDuration="4.34658165s" totalImagesPullingTime="2.710090182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:35.345588559 +0000 UTC m=+70.133248263" watchObservedRunningTime="2026-11-02 17:40:35.34658165 +0000 UTC m=+70.134241357" Nov 02 17:40:35.411832 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:35.411734 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.419653102 podStartE2EDuration="7.41171346s" totalImagesPullingTime="3.979565236s" totalInitContainerRuntime="12.495122ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:35.386312475 +0000 UTC m=+70.173972178" watchObservedRunningTime="2026-11-02 17:40:35.41171346 +0000 UTC m=+70.199373167" Nov 02 17:40:36.365494 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:36.365458 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" event={"ID":"4a7e0208-a8bc-43f2-8921-5f3d0909120a","Type":"ContainerStarted","Data":"7a2342e0e526f850cb69526920dba52f30ad1aea36cac50a05324950f6c4aeab"} Nov 02 17:40:36.365936 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:36.365518 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" event={"ID":"4a7e0208-a8bc-43f2-8921-5f3d0909120a","Type":"ContainerStarted","Data":"2429dd506b9dcc1761b65cbf311e33d83582738b043e9d9f2499ba500eb698ca"} Nov 02 17:40:36.365936 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:36.365536 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" event={"ID":"4a7e0208-a8bc-43f2-8921-5f3d0909120a","Type":"ContainerStarted","Data":"d946bd0b51fafdffc70f877b96f70f30e355d552ac7068a11744ab0bd4e21954"} Nov 02 17:40:36.394925 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:36.394872 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" podStartSLOduration=1.329617511 podStartE2EDuration="4.394855483s" totalImagesPullingTime="3.065237972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:36.392988478 +0000 UTC m=+71.180648183" watchObservedRunningTime="2026-11-02 17:40:36.394855483 +0000 UTC m=+71.182515187" Nov 02 17:40:38.377007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:38.376130 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89pdp" event={"ID":"3a654875-bb1c-4b4e-834d-9e8001f2a351","Type":"ContainerStarted","Data":"b652ca1246578bb22f2e70a633ccbd93e1b6f81daf416064b36f144b0a624632"} Nov 02 17:40:38.377007 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:38.376945 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:39.385071 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:39.384821 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"72ee5e4a15eda0d55245ac86207d753278932acb1b82debffe4284fc64b205d2"} Nov 02 17:40:39.385071 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:39.384951 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"09d406e6e8aee219d4de6693756da21712b3dd2ba319dda0b5e4ed69209effe1"} Nov 02 17:40:39.844924 ip-10-0-141-84 kubenswrapper[2581]: I1102 17:40:39.844861 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:40:39.845105 ip-10-0-141-84 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:40:39.858473 ip-10-0-141-84 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:40:39.858693 ip-10-0-141-84 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:40:39.858750 ip-10-0-141-84 systemd[1]: kubelet.service: Consumed 4.287s CPU time. Nov 02 17:40:39.870726 ip-10-0-141-84 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:40:39.929144 ip-10-0-141-84 kubenswrapper[6343]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:40:39.929144 ip-10-0-141-84 kubenswrapper[6343]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:40:39.929144 ip-10-0-141-84 kubenswrapper[6343]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:40:39.929144 ip-10-0-141-84 kubenswrapper[6343]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:40:39.931092 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931007 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.931092 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931091 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931096 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931100 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931103 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931107 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931111 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931114 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931117 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931120 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931122 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931125 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931127 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931130 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931133 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931135 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931138 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931141 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931143 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931146 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931154 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.931158 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931156 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931159 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931163 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931166 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931168 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931171 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931174 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931177 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931179 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931182 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931185 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931187 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931190 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931192 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931195 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931197 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931200 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931202 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931206 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.931637 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931208 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931211 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931213 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931215 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931218 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931220 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931223 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931226 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931228 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931231 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931233 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931236 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931239 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931244 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931247 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931249 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931252 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931255 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931257 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931259 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.932103 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931262 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931265 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931267 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931270 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931273 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931275 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931279 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931281 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931284 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931287 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931289 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931292 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931295 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931298 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931301 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931304 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931306 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931309 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931311 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.932569 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931315 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931319 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931322 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931325 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931328 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931331 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931334 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931337 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931340 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931342 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931345 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931347 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931350 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931353 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931355 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931358 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931360 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931363 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931365 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.933005 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931368 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931371 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931373 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931376 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931378 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931381 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931386 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.931391 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932243 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932251 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932254 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932257 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932260 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932262 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932265 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932268 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932271 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932274 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932276 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.933455 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932280 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932283 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932285 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932289 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932291 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932294 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932297 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932299 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932302 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932305 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932307 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932310 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932312 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932315 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932317 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932320 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932322 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932325 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932327 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.933943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932330 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932332 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932341 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932344 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932346 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932349 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932351 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932354 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932356 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932359 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932361 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932364 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932367 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932370 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932374 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932378 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932381 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932384 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932387 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.934401 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932389 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932392 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932394 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932397 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932399 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932403 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932405 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932408 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932411 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932413 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932416 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932419 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932421 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932424 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932426 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932429 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932436 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932439 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932442 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932445 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.934853 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932448 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932451 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932453 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932463 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932466 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932469 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932471 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932474 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932477 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932480 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932483 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932486 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932488 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932491 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932493 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932496 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932498 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932501 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932504 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932507 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.935403 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932509 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932511 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932514 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932516 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932519 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932521 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932524 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932527 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932529 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932537 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932540 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932542 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932545 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932548 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932550 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932555 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.932560 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933040 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933062 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.935885 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933067 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933070 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933074 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933078 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933081 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933084 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933086 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933089 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933092 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933094 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933097 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933099 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933102 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933105 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933113 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933115 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933118 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933120 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933123 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933126 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.936352 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933129 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933131 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933135 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933137 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933140 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933143 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933145 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933148 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933151 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933153 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933156 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933158 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933161 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933163 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933166 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933168 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933171 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933175 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933179 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.936875 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933182 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933185 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933187 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933190 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933192 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933195 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933198 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933202 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933205 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933208 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933211 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933213 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933216 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933218 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933221 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933223 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933226 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933229 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933231 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.937364 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933234 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933236 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933239 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933241 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933244 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933246 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933249 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933252 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933254 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933257 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933259 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933262 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933264 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933267 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933269 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933272 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933274 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933277 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933279 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933282 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.937806 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933285 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933288 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933291 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933294 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933296 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933299 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933301 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933304 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933307 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933309 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933313 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933315 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933318 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933320 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933323 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933326 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933328 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933331 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933333 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933335 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.938295 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933339 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933341 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933344 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933347 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933351 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.933354 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933450 6343 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933468 6343 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933474 6343 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933479 6343 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933483 6343 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933487 6343 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933493 6343 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933497 6343 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933500 6343 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933503 6343 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933507 6343 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933510 6343 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933513 6343 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933516 6343 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933519 6343 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933522 6343 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933525 6343 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:40:39.939014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933529 6343 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933533 6343 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933536 6343 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933539 6343 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933542 6343 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933545 6343 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933549 6343 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933552 6343 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933555 6343 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933558 6343 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933561 6343 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933564 6343 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933567 6343 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933571 6343 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933574 6343 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933578 6343 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933581 6343 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933584 6343 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933587 6343 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933590 6343 flags.go:64] FLAG: --enable-server="true" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933593 6343 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933598 6343 flags.go:64] FLAG: --event-burst="100" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933601 6343 flags.go:64] FLAG: --event-qps="50" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933604 6343 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:40:39.939731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933607 6343 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933610 6343 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933614 6343 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933618 6343 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933621 6343 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933624 6343 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933627 6343 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933630 6343 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933633 6343 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933636 6343 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933639 6343 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933642 6343 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933645 6343 flags.go:64] FLAG: --feature-gates="" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933649 6343 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933652 6343 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933655 6343 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933658 6343 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933662 6343 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933665 6343 flags.go:64] FLAG: --help="false" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933668 6343 flags.go:64] FLAG: --hostname-override="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933671 6343 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933675 6343 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933678 6343 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:40:39.940314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933681 6343 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933685 6343 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933688 6343 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933691 6343 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933694 6343 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933697 6343 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933702 6343 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933705 6343 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933708 6343 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933711 6343 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933714 6343 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933717 6343 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933720 6343 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933722 6343 flags.go:64] FLAG: --lock-file="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933725 6343 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933728 6343 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933731 6343 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933737 6343 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933740 6343 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933743 6343 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933746 6343 flags.go:64] FLAG: --logging-format="text" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933749 6343 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933753 6343 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933756 6343 flags.go:64] FLAG: --manifest-url="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933759 6343 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:40:39.940843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933763 6343 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933766 6343 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933770 6343 flags.go:64] FLAG: --max-pods="110" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933773 6343 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933776 6343 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933780 6343 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933782 6343 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933785 6343 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933788 6343 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933791 6343 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933798 6343 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933801 6343 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933804 6343 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933808 6343 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933811 6343 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933814 6343 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933817 6343 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933819 6343 flags.go:64] FLAG: --port="10250" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933823 6343 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933826 6343 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0e8575bfc32139708" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933829 6343 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933832 6343 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933835 6343 flags.go:64] FLAG: --register-node="true" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933837 6343 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933841 6343 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933844 6343 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:40:39.941463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933847 6343 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933850 6343 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933854 6343 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933857 6343 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933860 6343 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933863 6343 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933866 6343 flags.go:64] FLAG: --runonce="false" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933869 6343 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933872 6343 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933875 6343 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933878 6343 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933881 6343 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933884 6343 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933887 6343 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933890 6343 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933893 6343 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933896 6343 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933899 6343 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933902 6343 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933906 6343 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933909 6343 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933912 6343 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933917 6343 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933920 6343 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933923 6343 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:40:39.942208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933927 6343 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933930 6343 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933933 6343 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933936 6343 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933939 6343 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933942 6343 flags.go:64] FLAG: --v="2" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933946 6343 flags.go:64] FLAG: --version="false" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933950 6343 flags.go:64] FLAG: --vmodule="" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933954 6343 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.933958 6343 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934126 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934131 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934134 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934139 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934141 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934144 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934146 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934149 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934151 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934154 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934157 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934159 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.942817 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934162 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934165 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934167 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934170 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934174 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934177 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934179 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934182 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934185 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934187 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934190 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934192 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934195 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934197 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934200 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934202 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934205 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934208 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934210 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934213 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.943410 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934216 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934219 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934221 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934225 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934228 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934231 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934233 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934236 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934238 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934241 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934243 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934246 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934248 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934251 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934255 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934258 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934263 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934266 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934268 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934271 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.944229 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934273 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934276 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934278 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934281 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934284 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934286 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934289 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934291 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934294 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934297 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934299 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934302 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934304 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934307 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934310 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934313 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934316 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934319 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934322 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934325 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.944768 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934327 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934330 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934332 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934336 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934340 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934343 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934346 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934348 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934352 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934355 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934357 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934360 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934363 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934365 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934368 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934371 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934373 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934376 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934378 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.945349 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934381 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934383 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934386 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934389 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934391 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934394 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934396 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934399 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934404 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934407 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934410 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934412 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934415 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934418 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934420 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.934446 6343 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:40:39.945857 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934480 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934484 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934487 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934490 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934492 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934495 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934499 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934502 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934504 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934507 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934510 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934513 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934515 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934518 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934520 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934523 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934526 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934528 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934531 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934533 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.946443 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934536 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934538 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934541 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934543 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934548 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934550 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934553 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934556 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934558 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934561 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934563 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934566 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934568 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934571 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934573 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934576 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934579 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934582 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934586 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934588 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.947019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934591 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934593 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934596 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934598 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934601 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934604 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934606 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934609 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934611 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934614 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934618 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934622 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934625 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934628 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934631 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934634 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934638 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934641 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934644 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934647 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.947594 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934649 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934652 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934655 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934657 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934660 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934662 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934665 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934668 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934670 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934673 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934677 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934679 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934682 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934684 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934687 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934691 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934694 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934697 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934699 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.948120 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934703 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934707 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934710 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934713 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934716 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934719 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934722 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934725 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934727 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934732 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934734 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934738 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934741 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934744 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934747 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934749 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934752 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934754 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934757 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.948595 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934759 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934762 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934764 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934767 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934771 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934774 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934777 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.934780 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.939010 6343 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.939029 6343 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.939067 6343 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.939075 6343 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939369 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939385 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939394 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939511 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939517 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939520 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939523 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.949064 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939526 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939532 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939538 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939543 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939547 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939549 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939552 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939555 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939558 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939561 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939564 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939567 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939569 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939572 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939575 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939577 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939580 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939583 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939586 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.949515 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939588 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939591 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939594 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939597 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939600 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939603 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939606 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939609 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939614 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939617 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939620 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939622 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939625 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939628 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939631 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939633 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939636 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939639 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939641 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939644 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.949963 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939647 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939650 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939653 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939655 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939659 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939662 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939664 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939667 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939669 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939672 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939675 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939678 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939680 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939683 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939686 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939689 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939692 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939695 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939697 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.950438 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939700 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939702 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939705 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939708 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939710 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939713 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939716 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939718 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939721 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939724 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939726 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939729 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939731 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939734 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939737 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939740 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939743 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939746 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939748 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939751 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.950899 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939754 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939756 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939759 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939762 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939764 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939767 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939769 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939773 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939777 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939781 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939784 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939786 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939789 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939791 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939794 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939796 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939799 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939801 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939804 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.951560 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939806 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.939809 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.939843 6343 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940002 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940007 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940010 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940013 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940016 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940022 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940026 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940029 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940031 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940034 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940037 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940039 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940042 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.952191 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940064 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940068 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940072 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940075 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940078 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940080 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940083 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940085 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940088 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940091 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940093 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940095 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940098 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940101 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940103 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940107 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940109 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940112 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940114 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.952573 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940117 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940119 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940122 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940124 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940127 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940130 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940134 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940137 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940139 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940142 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940144 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940147 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940150 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940154 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940157 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940159 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940162 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940164 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940167 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.953012 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940169 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940172 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940175 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940177 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940180 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940182 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940185 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940187 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940190 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940192 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940195 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940199 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940203 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940206 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940209 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940212 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940214 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940217 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940220 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.953549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940222 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940225 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940228 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940230 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940233 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940235 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940238 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940241 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940243 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940246 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940248 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940251 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940253 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940256 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940258 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940261 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940263 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940266 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940268 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.954153 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940272 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940274 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940277 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940279 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940282 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940284 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940287 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940289 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940292 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940295 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940297 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940300 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940302 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940305 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940307 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940311 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.954903 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940313 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.940338 6343 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940368 6343 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940372 6343 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940375 6343 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940379 6343 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940381 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940384 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940387 6343 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940389 6343 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940392 6343 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940395 6343 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940397 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940400 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940402 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940405 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:39.955545 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940407 6343 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940410 6343 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940412 6343 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940415 6343 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940418 6343 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940420 6343 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940423 6343 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940426 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940430 6343 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940433 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940436 6343 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940439 6343 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940441 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940444 6343 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940446 6343 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940449 6343 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940452 6343 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940454 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940457 6343 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:39.955942 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940459 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940462 6343 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940465 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940467 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940469 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940472 6343 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940474 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940478 6343 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940481 6343 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940485 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940487 6343 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940490 6343 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940493 6343 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940495 6343 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940498 6343 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940500 6343 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940503 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940505 6343 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940508 6343 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:39.956521 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940510 6343 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940513 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940516 6343 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940518 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940521 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940523 6343 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940526 6343 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940528 6343 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940531 6343 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940533 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940536 6343 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940539 6343 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940541 6343 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940545 6343 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940547 6343 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940550 6343 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940552 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940554 6343 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940560 6343 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:39.956970 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940562 6343 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940565 6343 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940569 6343 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940571 6343 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940574 6343 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940576 6343 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940579 6343 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940582 6343 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940584 6343 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940587 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940589 6343 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940592 6343 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940594 6343 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940597 6343 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940600 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940602 6343 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940605 6343 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940608 6343 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940610 6343 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940613 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:39.957465 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940616 6343 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940619 6343 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940622 6343 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940624 6343 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940627 6343 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940631 6343 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940634 6343 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940637 6343 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940640 6343 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940642 6343 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940645 6343 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940649 6343 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940651 6343 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940653 6343 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:40:39.940657 6343 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.940711 6343 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.941620 6343 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.941670 6343 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.941836 6343 server.go:1014] "Starting client certificate rotation" Nov 02 17:40:39.957943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.941987 6343 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.942063 6343 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:34:25 +0000 UTC" deadline="2028-04-28 05:57:50.126679721 +0000 UTC" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.942102 6343 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13020h17m10.18458075s" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.942297 6343 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.943297 6343 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.944595 6343 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.946481 6343 log.go:25] "Validated CRI v1 image API" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.947287 6343 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:40:39.958409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.949372 6343 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9ef89d6b-bd4e-4ff3-afcf-d988820f4553:/dev/nvme0n1p3 cfdb2b64-95f8-4df4-80e2-4db4600b2d98:/dev/nvme0n1p4] Nov 02 17:40:39.958630 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.949392 6343 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0/userdata/shm major:0 minor:428 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba/userdata/shm major:0 minor:266 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e70cbc169df2a25beb706b74739c8e7a568a56c90863c60ad115517f354ed62d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e70cbc169df2a25beb706b74739c8e7a568a56c90863c60ad115517f354ed62d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/094f3261-e65e-4fd0-893d-0e9c0f99988f/volumes/kubernetes.io~projected/kube-api-access-pxwnb:{mountpoint:/var/lib/kubelet/pods/094f3261-e65e-4fd0-893d-0e9c0f99988f/volumes/kubernetes.io~projected/kube-api-access-pxwnb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12700290-2871-451f-9f86-3ea2830de15d/volumes/kubernetes.io~projected/kube-api-access-2p7f4:{mountpoint:/var/lib/kubelet/pods/12700290-2871-451f-9f86-3ea2830de15d/volumes/kubernetes.io~projected/kube-api-access-2p7f4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~projected/kube-api-access-92r8b:{mountpoint:/var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~projected/kube-api-access-92r8b major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20c5900d-985b-4db3-8232-5c24ff85cc01/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/20c5900d-985b-4db3-8232-5c24ff85cc01/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~projected/kube-api-access-4pskb:{mountpoint:/var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~projected/kube-api-access-4pskb major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a654875-bb1c-4b4e-834d-9e8001f2a351/volumes/kubernetes.io~projected/kube-api-access-8sl6f:{mountpoint:/var/lib/kubelet/pods/3a654875-bb1c-4b4e-834d-9e8001f2a351/volumes/kubernetes.io~projected/kube-api-access-8sl6f major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40498a67-6c10-479b-b130-3c97cdd76774/volumes/kubernetes.io~projected/kube-api-access-xvnjw:{mountpoint:/var/lib/kubelet/pods/40498a67-6c10-479b-b130-3c97cdd76774/volumes/kubernetes.io~projected/kube-api-access-xvnjw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40498a67-6c10-479b-b130-3c97cdd76774/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/40498a67-6c10-479b-b130-3c97cdd76774/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~projected/kube-api-access-45684:{mountpoint:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~projected/kube-api-access-45684 major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~projected/kube-api-access-lpbdb:{mountpoint:/var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~projected/kube-api-access-lpbdb major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/volumes/kubernetes.io~projected/kube-api-access-jj4fr:{mountpoint:/var/lib/kubelet/pods/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/volumes/kubernetes.io~projected/kube-api-access-jj4fr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/volumes/kubernetes.io~secret/metrics-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/config-reloader/2 major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/prometheus/15 major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~empty-dir/config-out major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~projected/kube-api-access-47lmk:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~projected/kube-api-access-47lmk major:0 minor:417 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~projected/tls-assets major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/config major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:413 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/web-config major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~projected/kube-api-access-9xj2v:{mountpoint:/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~projected/kube-api-access-9xj2v major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68c1bf64-b418-49c2-9753-4fad4bff08ae/volumes/kubernetes.io~projected/kube-api-access-2f6zm:{mountpoint:/var/lib/kubelet/pods/68c1bf64-b418-49c2-9753-4fad4bff08ae/volumes/kubernetes.io~projected/kube-api-access-2f6zm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72cc37d8-2509-4902-8494-415871c9db22/volumes/kubernetes.io~projected/kube-api-access-kn6q6:{mountpoint:/var/lib/kubelet/pods/72cc37d8-2509-4902-8494-415871c9db22/volumes/kubernetes.io~projected/kube-api-access-kn6q6 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72cc37d8-2509-4902-8494-415871c9db22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/72cc37d8-2509-4902-8494-415871c9db22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8271b561-5608-4f0c-93eb-0850dcdcd9c6/volumes/kubernetes.io~projected/kube-api-access-gqmqr:{mountpoint:/var/lib/kubelet/pods/8271b561-5608-4f0c-93eb-0850dcdcd9c6/volumes/kubernetes.io~projected/kube-api-access-gqmqr major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volumes/kubernetes.io~projected/kube-api-access-9pr69:{mountpoint:/var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volumes/kubernetes.io~projected/kube-api-access-9pr69 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e998028-a165-4d2b-8c14-c422d8a4d099/volumes/kubernetes.io~projected/kube-api-access-lkdgd:{mountpoint:/var/lib/kubelet/pods/9e998028-a165-4d2b-8c14-c422d8a4d099/volumes/kubernetes.io~projected/kube-api-access-lkdgd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e998028-a165-4d2b-8c14-c422d8a4d099/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9e998028-a165-4d2b-8c14-c422d8a4d099/volumes/kubernetes.io~secret/metrics-certs major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1/volumes/kubernetes.io~projected/kube-api-access-b4n6r:{mountpoint:/var/lib/kubelet/pods/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1/volumes/kubernetes.io~projected/kube-api-access-b4n6r major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~projected/kube-api-access-2fxvz:{mountpoint:/var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~projected/kube-api-access-2fxvz major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~projected/kube-api-access-sbqvg:{mountpoint:/var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~projected/kube-api-access-sbqvg major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd8dc667-bb79-4137-845c-4cff317422ed/volumes/kubernetes.io~projected/kube-api-access-mbml9:{mountpoint:/var/lib/kubelet/pods/bd8dc667-bb79-4137-845c-4cff317422ed/volumes/kubernetes.io~projected/kube-api-access-mbml9 major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd8dc667-bb79-4137-845c-4cff317422ed/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bd8dc667-bb79-4137-845c-4cff317422ed/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee56fd64-8479-45f4-b098-16208b7ced9f/volumes/kubernetes.io~projected/kube-api-access-zflxg:{mountpoint:/var/lib/kubelet/pods/ee56fd64-8479-45f4-b098-16208b7ced9f/volumes/kubernetes.io~projected/kube-api-access-zflxg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/alertmanager/9 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/config-reloader/6 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~empty-dir/config-out major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~projected/kube-api-access-6lpmm:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~projected/kube-api-access-6lpmm major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~projected/tls-assets major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/config-volume major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/web-config major:0 minor:310 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/695d33e9846a7ef930a1a5a01108483907faf53455576928497eb137df404e82/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e18edd6f88eb0bbb5407b9da0dfe6b5190a03c4f3973e6103e7ad90bb4cb0e81/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e232f89aa79650b1c322e27cdb868fab91245fed1d77204c5df3b08e4904ab3a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/191590dd3eb0afcd489f5cf7f02b9602c22da957efcbbe81393101f4e2a58c13/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/97043b73f729c9c737c83bbec8dfc72cdb49f427f71f8a2481434350bfa84d4b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/59d1958e5fec29f470e65e3fdc659116e8d2f04604ae730911db783f7f65dd5a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/54484d8c6dad267f79559326a895b0be792db61c7eb999eefed315283ba04e9a/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8f6746bdef92d4239783afa57f844f84804e87620012b08735ed58dda15dcc78/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1fa89a737b4f1c509325741356c32ab6447813e5f14c8a83522959467e835a6e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/74a5da330cd2677fce332d086ae5bef0bb9614d1bdd8c9d0eae0eb0b36fbf343/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5f1e963710cd804a09aaeaec35b85b124448dc7aa7cf64a3da71065623ec9476/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/06ab0cb69c4d15298801537db59a7ea864accc604edb6a6cb2aa48bdb3dfaebf/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/49dade1ac08ba5f1501a577121e7cb02a697fe8fd8eaf4af6d96f1bb829fd203/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e6d512d71ebcdbdb771870f07bbe86dd099fa2cb7ad747a2610a74a9641f619a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/a425190c817c28fc799db9d99b6a884f4984e214f7f82ad4865500c1539b1dd3/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d0a4f723190f934390b3bdb70e697a19f982880d322d6e9d78e818f89abfffc0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b732d226b713558e5924f3a7eb3ac372a4dde03c595d4e3dc289c034050aae1c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e1af85dacb01516209b2e07b1b939c86738e636591cb7f511bbcc03df14a7ef4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1ea26994697f6144e66407bf69f07ad8c106534dd718f14243b8c0847dcbc3b0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/483fe25d3d02d268949dfd3dfbae0cbdedbd61ab1c9019ae745a5796b5b0be16/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b8508a0dd0793d20d7a2674cdfcbba950b0d2d517e1492f0e4018027fa3f2b2d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/cc3a42686e9c45cfcaad1a4cd385942fcca68353d33a565412b05e9a9422ff9e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/fafca753a17d27e33079a5a810aa407be3cd9c9103541b97e387335f71f70a25/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/7d90f1fe52ca7c6a170d2c2c0edee8a27afc21b967fb758f6dc6d8bd27d1869b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/baff3408a33ce2358b1f5978540feaa7f03c94942291de24ca86a8ba483abb17/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/02fb0262f96e0dc28dd10a52fff5589aafc7008204b334efdcc6d5dadcb339e8/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8b87044988c9ae3cb78582f9ba5a108dd054f8ed11c78cd658cc3b9d3e865030/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/7f7ac4118ad9d9bcf1ae922cf6092b7f65fde129012c746bd84c43bce979a854/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/acabd07afa0f9ac07bb0b85f34837adf4177fe01f90fe4b9eb54a9f483642afd/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/7e6b375b66181e6b5e94a4db686f80607dbb14981487da26ecf2c2225ed927b5/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6a62ba99e2ee9b6da58c9709fd0f02fd168c60cf7d8692a8e6489ea446cef68f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/272de3dafe86b112ccf065d2ff4f7374ce0a36461e6c5f89c5a802c02576d919/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/4902b8314b6f5a70a0491003e7d49390dcdb471c9b378792e19f3394853b941e/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/b222e318dc4dc9b1e9d02eef596141352b97a021414ce43f04d090abe4eeba70/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/fe1627bc7684532c34117ad9f84fcac752e2d443f0b09dc8637ea2149e2f6b94/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/6d01e6e83e3a8a6ff34ecf23b256ed8217ce0e85cc2b7e59d49e511496cdec05/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1f859b3270e8d9b0e6dab67333ce1a27f13f5f6b371796134eaec9d34102fc9c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/7af10d61f9b5c4061834488c40b2513d23c045b351352873f48d0b680d2cebc7/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/5be82b2d2e1f37641c22a4e0f1d74f1ff87e80eabc82d72e738a1cd32fb0edd4/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0f69ab12e65cab6d31b06542518c4c20b1032af2446dd83dd590b1b63dee23a2/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/6ff919e08e46f3f3efec64473e3c9e10e5a3fc22cee9017c68f97e4715caae69/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/62777f36a3a7ed79eab5d490f9ad6e45ff5106566ca898c5e392c7581f1c8791/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/61c116929f4aaf38c9f7ed385c41bd252c432101d1a78dbb03f92a5efb79c359/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/5f894c0f1b94222c6af2b6bbc06f64dae67edfd9eb8cdaf6bbe614720db7bc02/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/1844c9b24c728defd85e262e7c01f10ff60ec28dc1b1ffe4ce6c9f90d2776841/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/2e9948063d7a36d876a21ba2ef94cf6568d4c37c264a7e84947fd5eeef5e514a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/d9e2d8eec4c6bb80e4b692e5b1066f65ecfffafe7f4591b36b9175800514be34/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/4d21efe20832096faf0ff04271de07debee151adb95aac39b3fa9eb32425cc84/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/4bb32ab76e730583d5fa5203b482537fcca2219be2edf17336457f9b5c19c0da/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/b6dea0f25486712f737dee5e2b10772a11c87e662b9c621f15e2bf6164cb720d/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/e465b6608972d2160327508a77c0c29c2ccedbea9d95c09e2102186121d26d92/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/b18c933eaec0ebf607ee7a06590629b612bc41d43dcb49f549794e1325780c3a/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/be9f6b539f40a89c810570f33962d2a8a0c43efe4a9b87c1c9cef7f16e9ef386/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/c28c3413f0d26b88ffca1dd97926a3a961ccf9a36ae3cde18634301256c86df7/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/42b738f685e4c5a3e42f1c8fb988feebe613e0222cd630151160f241d853746b/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/a8119aee0c1664bfc9f93aec54e72f408a93eaf8e3f7996516b9d9b5840659cf/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/ef313c526e1b79261e22d58f69b394519b17825a739c87ad5c0e33ab68221694/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/eeebb70a9918ec7d5623ced1b82acb0b94b60b0bebeb176841b3a045595f9b62/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/ad95c346c75cdff78583cf8be208463c80d771b78bc0cf4b73f112cf072c7bb5/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/8413ab6277518c2202c13594841ea4741e38623b29ed14b6d3b8bc4514cb5341/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/2d04d836bdd86073c0626b7e8b38919ac082f7f2317fb33e27b3ce688ab57546/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/972deb40409d80c2ba7275b9cc78ab80e284ccc4792d4694be39a9078a0dcc94/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/0032701b465111fc816b72ac027f7a8e6a95057aef84a5d2c72e5a4e9a9e52e7/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/0a5b76e0210ef7087d13aa6366ab0e0d47f11f08b1123736a6def284ead4e74f/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/2c4f4026712612650258070847f6ada6911ab6d30f5810156864eedfd9b6f199/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/d58570fc91cc52e4ad6106ce84c3c0d188c0face94211ffc6be45f039d9f792b/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/f612b5db30cec855e3f09b2725fcc27ca88cb2a0e12b7a250d0451580aa07dae/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/3cb5fddde806871138315e0c6631e022637249cfdd0bb3538c748afdc66ac8f8/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/11403d2ddb84094baba77a89d9ff19466d21d28386ec4b36e94bfd5ed9b5f9f9/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/abf5df3412316c53be961526edfd5bc0f96ebe36a1389e1d37cff1472490dd17/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fe2b637d7f99b131eaf9933849bddadd38d47a75b7ffc33639a662498b2553b3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/de72b594e86a1f464bae9a1ec9b38f27d661309261ed9f61ab6c43779b12ffb7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/603b8ed1888339157024bcb9f6273e77dadbd86138abe00936d162ecc59c4755/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/dc5b22f45a62b60931c95cd274fc9d4ea7a53fb3fd17e3eee905d3fcb3f8d466/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7ca71d0d83496b1108d97b4410598a29a1882c974fbe457a238843725d612fb9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9a082ce3be404b16d454ce3db86a9119f7852e3b58e0658d7e1a909aeabc67e5/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/be4e0a51feda6ba462d0d531596c452300b422e4eaba17a52e4843bcccdfa92b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/5a596733d7341c62764b780cf7e91d331b66041cf4473e35739152aaa2c01c63/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1b77a3634d87734632a505e69aceec52b1fcd8179f5ecdd258352dc212314107/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 17:40:39.958952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.958493 6343 manager.go:217] Machine: {Timestamp:2026-11-02 17:40:39.958028595 +0000 UTC m=+0.057056783 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120532 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae62e3970f6bdcbba9a13a2e3beba SystemUUID:ec2ae62e-3970-f6bd-cbba-9a13a2e3beba BootID:8c8807dd-fe56-48f0-bfcc-e702f22a8c62 Filesystems:[{Device:/run/containers/storage/overlay-containers/5ab5118650b8483948fa324ad01d6cab379301280c43525d9e19a5592e7fc52e/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c538a037f33c70dd59c1de8fbd2ce3eea80546231a031921fd00816b5b7ec59/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97eae7806d43acba37b775d7ea453dbcfd0e31ac54fc954dd0eaa06e5037e316/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1/volumes/kubernetes.io~projected/kube-api-access-b4n6r DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:414 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:415 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40498a67-6c10-479b-b130-3c97cdd76774/volumes/kubernetes.io~projected/kube-api-access-xvnjw DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12700290-2871-451f-9f86-3ea2830de15d/volumes/kubernetes.io~projected/kube-api-access-2p7f4 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~projected/kube-api-access-92r8b DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d90f056344ecadb86e70e396aa2246fda4b4b8c99f23448e042cbf611442798c/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/315c8dfc712eb1b9a91b59f3260dab17e73194340e77c20d7e21565ee1637f12/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:413 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bd8dc667-bb79-4137-845c-4cff317422ed/volumes/kubernetes.io~projected/kube-api-access-mbml9 DeviceMajor:0 DeviceMinor:370 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~projected/kube-api-access-6lpmm DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/98c18ffc-40f4-4360-9095-b08f95cab47c/volumes/kubernetes.io~projected/kube-api-access-9pr69 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0/userdata/shm DeviceMajor:0 DeviceMinor:428 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7027de9ded99b16727ee3dcacaeb7f4285f729222806f31da44f8430d9e93d0e/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/72cc37d8-2509-4902-8494-415871c9db22/volumes/kubernetes.io~projected/kube-api-access-kn6q6 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e70cbc169df2a25beb706b74739c8e7a568a56c90863c60ad115517f354ed62d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~projected/kube-api-access-lpbdb DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2145837e07f1837229bb06ab94444e17aee2d64fdde89f62558b1661a53e8f89/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:359 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:418 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/258f066558bde0433439cb3b98c99b6d1c2a85f42cef6fcab689c97e9d2613ef/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2aa794e61d6ed5c63b930739720680d65ed87dd330b6f23ae266d0fd5573ba1e/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:414 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/68c1bf64-b418-49c2-9753-4fad4bff08ae/volumes/kubernetes.io~projected/kube-api-access-2f6zm DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f32e2efd1dd877983df7a8afc7add49bdad65e1514250ccb1839ce577d4b3f8f/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/20c5900d-985b-4db3-8232-5c24ff85cc01/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1406834031e039e3c631836561ed0b3e05bebf0e0d27edd9e17e1ad63c0a4b87/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9005c0a3087b864dc797510f38b5aae467d86679fbe6ebc68978d19948ac8bba/userdata/shm DeviceMajor:0 DeviceMinor:266 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~projected/kube-api-access-sbqvg DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bd8dc667-bb79-4137-845c-4cff317422ed/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:369 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:379 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:411 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~projected/kube-api-access-2fxvz DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~projected/kube-api-access-4pskb DeviceMajor:0 DeviceMinor:264 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/094f3261-e65e-4fd0-893d-0e9c0f99988f/volumes/kubernetes.io~projected/kube-api-access-pxwnb DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fddeb6f237765d7b6aaaaa38551ca639f06b80daa091f9b9386f55867286852/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5701f7c4-97ba-4887-b9fe-4e3bc9941acb/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d22e1a30a1e2cded0e9082a276877348689f747834b5f7ab60c93c1ceefdced9/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09e20600d0f97ad6121e4f7a2c6ce9865ce1f01bb0d252463eb7c0d738ea7827/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bd51ff1f728ed47940b05c7a138bdf9a051d67b101756e1250143bfcc775bbd/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:407 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:414 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee56fd64-8479-45f4-b098-16208b7ced9f/volumes/kubernetes.io~projected/kube-api-access-zflxg DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a654875-bb1c-4b4e-834d-9e8001f2a351/volumes/kubernetes.io~projected/kube-api-access-8sl6f DeviceMajor:0 DeviceMinor:360 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~projected/kube-api-access-45684 DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9e998028-a165-4d2b-8c14-c422d8a4d099/volumes/kubernetes.io~projected/kube-api-access-lkdgd DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aec52e3c-2a94-48a7-b568-ab54be3ccb9e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:414 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/volumes/kubernetes.io~projected/kube-api-access-jj4fr DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a589c94416d7e358f25e17d928a6d7026b13761f8f6c21b96a31f82accb9309/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~projected/kube-api-access-9xj2v DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5a5d720-67fb-459f-823f-ad9993dec96c/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a257c61d1bef5c04bc7021a3d7ae18e34789050f5d68b786a32a18d3dade8848/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:416 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58d8c6d173d06b20ecae239bdb0c39f18c3dfda0fc1adbcb826aaa090d316a78/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40498a67-6c10-479b-b130-3c97cdd76774/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7e0208-a8bc-43f2-8921-5f3d0909120a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:380 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~projected/kube-api-access-47lmk DeviceMajor:0 DeviceMinor:417 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8271b561-5608-4f0c-93eb-0850dcdcd9c6/volumes/kubernetes.io~projected/kube-api-access-gqmqr DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e998028-a165-4d2b-8c14-c422d8a4d099/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72cc37d8-2509-4902-8494-415871c9db22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:409 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31b1e410-3fff-449a-a344-8ca439c0c9af/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:408 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:412 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:09e20600d0f97ad MacAddress:3a:49:dc:29:90:a2 Speed:10000 Mtu:8901} {Name:1bd51ff1f728ed4 MacAddress:3e:0c:1a:95:9d:c9 Speed:10000 Mtu:8901} {Name:2145837e07f1837 MacAddress:f6:15:71:33:c6:6d Speed:10000 Mtu:8901} {Name:2aa794e61d6ed5c MacAddress:82:43:2a:7e:09:63 Speed:10000 Mtu:8901} {Name:51b046ef081e98d MacAddress:72:81:c6:c5:54:ca Speed:10000 Mtu:8901} {Name:56090f1566ebea9 MacAddress:8e:3f:1d:f6:fd:6c Speed:10000 Mtu:8901} {Name:85248e95a92d245 MacAddress:8e:18:37:88:78:59 Speed:10000 Mtu:8901} {Name:9005c0a3087b864 MacAddress:ce:3d:19:8c:8c:0a Speed:10000 Mtu:8901} {Name:97eae7806d43acb MacAddress:76:75:3d:c0:8a:87 Speed:10000 Mtu:8901} {Name:9a589c94416d7e3 MacAddress:86:05:f3:73:2b:c3 Speed:10000 Mtu:8901} {Name:a257c61d1bef5c0 MacAddress:32:39:09:58:46:40 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:67:68:89 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:49:85:94:bf:1c Speed:0 Mtu:8901} {Name:d22e1a30a1e2cde MacAddress:f6:6e:3e:93:bb:52 Speed:10000 Mtu:8901} {Name:d90f056344ecadb MacAddress:a6:97:94:b4:9f:ed Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:67:68:89 Speed:0 Mtu:9001} {Name:f32e2efd1dd8779 MacAddress:5a:9a:0a:10:c5:6e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b6:19:19:74:04:a3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:7d:34:6c:be:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:40:39.959282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.958956 6343 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:40:39.959282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.958999 6343 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:40:39.959392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959348 6343 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:40:39.959523 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959376 6343 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-84.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:40:39.959595 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959525 6343 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:40:39.959595 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959538 6343 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:40:39.959595 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959551 6343 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:40:39.959595 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959578 6343 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:40:39.959774 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959671 6343 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:40:39.959888 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959874 6343 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:40:39.960011 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.959999 6343 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:40:39.960094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960065 6343 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:40:39.960094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960084 6343 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:40:39.960190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960099 6343 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:40:39.960190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960114 6343 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:40:39.960190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960126 6343 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:40:39.960777 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960761 6343 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:40:39.960777 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.960776 6343 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:40:39.961262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.961246 6343 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:40:39.974643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.974624 6343 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:39.978811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.977984 6343 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:39.980291 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980276 6343 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:40:39.980291 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980291 6343 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:40:39.980412 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980401 6343 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:40:39.980592 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980581 6343 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:40:39.980622 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980611 6343 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:40:39.980735 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980725 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:40:39.980768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980741 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:40:39.980768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980747 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:40:39.980768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980753 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:40:39.980768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980758 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:40:39.980768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980764 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:40:39.980768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980770 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980776 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980783 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980792 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980804 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980814 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980841 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:40:39.980983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.980847 6343 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:40:39.984243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.984229 6343 server.go:1263] "Started kubelet" Nov 02 17:40:39.984329 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.984272 6343 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:40:39.984690 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.984643 6343 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:40:39.984762 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.984710 6343 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:40:39.984716 ip-10-0-141-84 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:40:39.984979 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.984963 6343 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:40:39.985727 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.985711 6343 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:40:39.988974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.988957 6343 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:40:39.989256 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.989239 6343 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:40:39.989388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.989374 6343 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:40:39.989388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.989388 6343 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:40:39.989598 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.989585 6343 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:40:39.990973 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.990464 6343 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:34:26 +0000 UTC" deadline="2028-04-04 15:48:05.743193929 +0000 UTC" Nov 02 17:40:39.991105 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.990974 6343 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12454h7m25.75222521s" Nov 02 17:40:39.993304 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993285 6343 factory.go:55] Registering systemd factory Nov 02 17:40:39.993304 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993307 6343 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:40:39.993741 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993726 6343 factory.go:153] Registering CRI-O factory Nov 02 17:40:39.993835 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993743 6343 factory.go:223] Registration of the crio container factory successfully Nov 02 17:40:39.993835 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993801 6343 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:40:39.993835 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993819 6343 factory.go:103] Registering Raw factory Nov 02 17:40:39.993835 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.993836 6343 manager.go:1199] Started watching for new ooms in manager Nov 02 17:40:39.994197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994161 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40498a67-6c10-479b-b130-3c97cdd76774" volumeName="kubernetes.io/secret/40498a67-6c10-479b-b130-3c97cdd76774-cert" seLinuxMountContext="" Nov 02 17:40:39.994285 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994201 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8271b561-5608-4f0c-93eb-0850dcdcd9c6" volumeName="kubernetes.io/projected/8271b561-5608-4f0c-93eb-0850dcdcd9c6-kube-api-access-gqmqr" seLinuxMountContext="" Nov 02 17:40:39.994285 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994222 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config" seLinuxMountContext="" Nov 02 17:40:39.994361 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994235 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out" seLinuxMountContext="" Nov 02 17:40:39.994444 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994351 6343 manager.go:319] Starting recovery of all containers Nov 02 17:40:39.994499 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994472 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 17:40:39.994551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994508 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 17:40:39.994551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994525 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72cc37d8-2509-4902-8494-415871c9db22" volumeName="kubernetes.io/empty-dir/72cc37d8-2509-4902-8494-415871c9db22-data-volume" seLinuxMountContext="" Nov 02 17:40:39.994551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994543 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.994691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994562 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out" seLinuxMountContext="" Nov 02 17:40:39.994691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994582 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets" seLinuxMountContext="" Nov 02 17:40:39.995160 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.994595 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 17:40:39.995235 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995171 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client" seLinuxMountContext="" Nov 02 17:40:39.995235 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995195 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5701f7c4-97ba-4887-b9fe-4e3bc9941acb" volumeName="kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.995235 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995216 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995237 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995256 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995268 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995316 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12700290-2871-451f-9f86-3ea2830de15d" volumeName="kubernetes.io/configmap/12700290-2871-451f-9f86-3ea2830de15d-serviceca" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995330 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31b1e410-3fff-449a-a344-8ca439c0c9af" volumeName="kubernetes.io/projected/31b1e410-3fff-449a-a344-8ca439c0c9af-kube-api-access-4pskb" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995342 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40498a67-6c10-479b-b130-3c97cdd76774" volumeName="kubernetes.io/projected/40498a67-6c10-479b-b130-3c97cdd76774-kube-api-access-xvnjw" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995360 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995372 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5701f7c4-97ba-4887-b9fe-4e3bc9941acb" volumeName="kubernetes.io/empty-dir/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-volume-directive-shadow" seLinuxMountContext="" Nov 02 17:40:39.995395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995390 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995402 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995415 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" volumeName="kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995432 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995445 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="094f3261-e65e-4fd0-893d-0e9c0f99988f" volumeName="kubernetes.io/projected/094f3261-e65e-4fd0-893d-0e9c0f99988f-kube-api-access-pxwnb" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995642 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12700290-2871-451f-9f86-3ea2830de15d" volumeName="kubernetes.io/projected/12700290-2871-451f-9f86-3ea2830de15d-kube-api-access-2p7f4" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995671 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-federate-client-tls" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995700 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6440b09a-ee4a-4ffa-8102-da0d1d3e10bb" volumeName="kubernetes.io/configmap/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-config-volume" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995713 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995729 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995743 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995756 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20c5900d-985b-4db3-8232-5c24ff85cc01" volumeName="kubernetes.io/secret/20c5900d-985b-4db3-8232-5c24ff85cc01-agent-certs" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995772 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68c1bf64-b418-49c2-9753-4fad4bff08ae" volumeName="kubernetes.io/projected/68c1bf64-b418-49c2-9753-4fad4bff08ae-kube-api-access-2f6zm" seLinuxMountContext="" Nov 02 17:40:39.995786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995785 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72cc37d8-2509-4902-8494-415871c9db22" volumeName="kubernetes.io/projected/72cc37d8-2509-4902-8494-415871c9db22-kube-api-access-kn6q6" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995816 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8271b561-5608-4f0c-93eb-0850dcdcd9c6" volumeName="kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-daemon-config" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995829 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bd74c07-03e4-44b3-b5d3-4f06a5acb51c" volumeName="kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995846 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995890 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5701f7c4-97ba-4887-b9fe-4e3bc9941acb" volumeName="kubernetes.io/projected/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-api-access-lpbdb" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.995904 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996177 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996207 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e998028-a165-4d2b-8c14-c422d8a4d099" volumeName="kubernetes.io/projected/9e998028-a165-4d2b-8c14-c422d8a4d099-kube-api-access-lkdgd" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996226 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e998028-a165-4d2b-8c14-c422d8a4d099" volumeName="kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996239 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" volumeName="kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996252 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" volumeName="kubernetes.io/empty-dir/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-textfile" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996269 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" volumeName="kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-tls" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996281 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a5d720-67fb-459f-823f-ad9993dec96c" volumeName="kubernetes.io/projected/a5a5d720-67fb-459f-823f-ad9993dec96c-kube-api-access-2fxvz" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996297 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" volumeName="kubernetes.io/projected/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-kube-api-access-sbqvg" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996309 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee56fd64-8479-45f4-b098-16208b7ced9f" volumeName="kubernetes.io/projected/ee56fd64-8479-45f4-b098-16208b7ced9f-kube-api-access-zflxg" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996322 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996340 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996353 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="094f3261-e65e-4fd0-893d-0e9c0f99988f" volumeName="kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:40:39.996373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996371 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="094f3261-e65e-4fd0-893d-0e9c0f99988f" volumeName="kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-binary-copy" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996385 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5701f7c4-97ba-4887-b9fe-4e3bc9941acb" volumeName="kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996398 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996443 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996456 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee56fd64-8479-45f4-b098-16208b7ced9f" volumeName="kubernetes.io/empty-dir/ee56fd64-8479-45f4-b098-16208b7ced9f-tmp-dir" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996468 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/projected/4a7e0208-a8bc-43f2-8921-5f3d0909120a-kube-api-access-45684" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996484 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68c1bf64-b418-49c2-9753-4fad4bff08ae" volumeName="kubernetes.io/configmap/68c1bf64-b418-49c2-9753-4fad4bff08ae-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996496 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" volumeName="kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996511 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd8dc667-bb79-4137-845c-4cff317422ed" volumeName="kubernetes.io/projected/bd8dc667-bb79-4137-845c-4cff317422ed-kube-api-access-mbml9" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996523 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996537 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996554 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996566 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c" volumeName="kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-tuned" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996582 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a654875-bb1c-4b4e-834d-9e8001f2a351" volumeName="kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996595 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996607 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-client-tls" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996622 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5701f7c4-97ba-4887-b9fe-4e3bc9941acb" volumeName="kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996635 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6440b09a-ee4a-4ffa-8102-da0d1d3e10bb" volumeName="kubernetes.io/empty-dir/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-tmp-dir" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996654 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="094f3261-e65e-4fd0-893d-0e9c0f99988f" volumeName="kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996667 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c" volumeName="kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-tmp" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996683 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31b1e410-3fff-449a-a344-8ca439c0c9af" volumeName="kubernetes.io/configmap/31b1e410-3fff-449a-a344-8ca439c0c9af-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996696 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996709 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72cc37d8-2509-4902-8494-415871c9db22" volumeName="kubernetes.io/configmap/72cc37d8-2509-4902-8494-415871c9db22-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996726 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c18ffc-40f4-4360-9095-b08f95cab47c" volumeName="kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-env-overrides" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996739 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c18ffc-40f4-4360-9095-b08f95cab47c" volumeName="kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996754 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c18ffc-40f4-4360-9095-b08f95cab47c" volumeName="kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996763 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996772 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996810 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c" volumeName="kubernetes.io/projected/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-kube-api-access-92r8b" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996819 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31b1e410-3fff-449a-a344-8ca439c0c9af" volumeName="kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996832 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31b1e410-3fff-449a-a344-8ca439c0c9af" volumeName="kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996858 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5701f7c4-97ba-4887-b9fe-4e3bc9941acb" volumeName="kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996871 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6440b09a-ee4a-4ffa-8102-da0d1d3e10bb" volumeName="kubernetes.io/secret/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-metrics-tls" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996891 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c18ffc-40f4-4360-9095-b08f95cab47c" volumeName="kubernetes.io/projected/98c18ffc-40f4-4360-9095-b08f95cab47c-kube-api-access-9pr69" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996903 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996913 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996921 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72cc37d8-2509-4902-8494-415871c9db22" volumeName="kubernetes.io/secret/72cc37d8-2509-4902-8494-415871c9db22-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996928 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8271b561-5608-4f0c-93eb-0850dcdcd9c6" volumeName="kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cni-binary-copy" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996938 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c18ffc-40f4-4360-9095-b08f95cab47c" volumeName="kubernetes.io/secret/98c18ffc-40f4-4360-9095-b08f95cab47c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996946 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a5d720-67fb-459f-823f-ad9993dec96c" volumeName="kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996956 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd8dc667-bb79-4137-845c-4cff317422ed" volumeName="kubernetes.io/secret/bd8dc667-bb79-4137-845c-4cff317422ed-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996964 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996972 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6440b09a-ee4a-4ffa-8102-da0d1d3e10bb" volumeName="kubernetes.io/projected/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-kube-api-access-jj4fr" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996983 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65bcced4-e7fa-44db-ba68-001da2189ec6" volumeName="kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.996991 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a34000-505d-44f9-95fc-2a1f3cb4e813" volumeName="kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.997001 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a5d720-67fb-459f-823f-ad9993dec96c" volumeName="kubernetes.io/configmap/a5a5d720-67fb-459f-823f-ad9993dec96c-metrics-client-ca" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.997010 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5a5d720-67fb-459f-823f-ad9993dec96c" volumeName="kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:40:39.996998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.997017 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:40:39.998963 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.997031 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume" seLinuxMountContext="" Nov 02 17:40:39.998963 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.997059 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config" seLinuxMountContext="" Nov 02 17:40:39.998963 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.998666 6343 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:40.000024 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:39.997077 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20c5900d-985b-4db3-8232-5c24ff85cc01" volumeName="kubernetes.io/configmap/20c5900d-985b-4db3-8232-5c24ff85cc01-konnectivity-ca" seLinuxMountContext="" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000030 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000066 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7e0208-a8bc-43f2-8921-5f3d0909120a" volumeName="kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000080 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1" volumeName="kubernetes.io/projected/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kube-api-access-b4n6r" seLinuxMountContext="" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000088 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets" seLinuxMountContext="" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000096 6343 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aaaba8-6e2f-4193-ac60-8f930a419e00" volumeName="kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000103 6343 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:40:40.000119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.000108 6343 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:40:40.003375 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.003358 6343 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:40:40.011084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.011029 6343 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:40:40.012356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.012340 6343 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:40:40.012356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.012357 6343 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:40:40.012462 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.012376 6343 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:40:40.012462 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:40.012411 6343 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:40:40.020152 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.020128 6343 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:40.021786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.021767 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:40:40.022968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.022944 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:40:40.023688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.023650 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed" exitCode=0 Nov 02 17:40:40.026251 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.026226 6343 generic.go:356] "Generic (PLEG): container finished" podID="aec52e3c-2a94-48a7-b568-ab54be3ccb9e" containerID="ab37125646669f2fc7bcd4aac79a7f2e06ae5c6bce637e5e1d0e68ff65a1cd98" exitCode=0 Nov 02 17:40:40.044016 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.043991 6343 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="a497b3362b624916aa0ce313175ccab1eed32ea1f96953f10fd39b7c82a2e88c" exitCode=0 Nov 02 17:40:40.044123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.044016 6343 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="e6d0c549701e0a5843b38caa84a257eb981b9cd60695de3baf73c70a08d0a309" exitCode=0 Nov 02 17:40:40.044123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.044027 6343 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="6b0019b580e913210111ee307d3e7996badedeb70ecf81cd05c7b928d5e12160" exitCode=0 Nov 02 17:40:40.044123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.044039 6343 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="7ac10d92712a459d316d83f8941fddc826bc7f88b8dd6b085ddccf1f160807ed" exitCode=0 Nov 02 17:40:40.044123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.044077 6343 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="9d313db15c562c99e70863e7a48c02d1c81fa911de931c97013d6f311b29978b" exitCode=0 Nov 02 17:40:40.044123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.044083 6343 generic.go:356] "Generic (PLEG): container finished" podID="094f3261-e65e-4fd0-893d-0e9c0f99988f" containerID="f333a5e18c68e69b3a3ea36255f6a59afb808533aebd6365524449b8b556884a" exitCode=0 Nov 02 17:40:40.049793 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.049774 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:40:40.050157 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.050137 6343 generic.go:356] "Generic (PLEG): container finished" podID="98c18ffc-40f4-4360-9095-b08f95cab47c" containerID="230a6af9e10a5bd19f3bc35436a4802c84ca9bfbf16084c97435e49a0cedec83" exitCode=1 Nov 02 17:40:40.075574 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.075540 6343 generic.go:356] "Generic (PLEG): container finished" podID="3f4115a2f4c3236862e1ddc68e0eda82" containerID="27b70c98a073f1bb30277b3b1bdee1a016de81c46af55f5dc7e8346a1e39e36a" exitCode=0 Nov 02 17:40:40.077792 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.077770 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="fb843d2f3022fb6c02b264443328dc82aa8488bf1ba72d318eb26ed758c057b4" exitCode=0 Nov 02 17:40:40.107018 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.107002 6343 manager.go:324] Recovery completed Nov 02 17:40:40.112459 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:40.112442 6343 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 17:40:40.130257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130236 6343 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:40:40.130257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130252 6343 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:40:40.130356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130272 6343 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:40:40.130400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130387 6343 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:40:40.130430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130397 6343 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:40:40.130430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130413 6343 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:40:40.130430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130418 6343 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:40:40.130430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130423 6343 policy_none.go:50] "Start" Nov 02 17:40:40.130430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130431 6343 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:40:40.130558 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130440 6343 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:40:40.130587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130560 6343 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:40:40.130587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130566 6343 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:40:40.130587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.130571 6343 policy_none.go:44] "Start" Nov 02 17:40:40.135882 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.135404 6343 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:40:40.141460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.141446 6343 manager.go:341] "Starting Device Plugin manager" Nov 02 17:40:40.141540 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:40.141475 6343 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:40:40.141540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.141487 6343 server.go:87] "Starting device plugin registration server" Nov 02 17:40:40.141788 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.141774 6343 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:40:40.141858 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.141788 6343 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:40:40.141906 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.141886 6343 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:40:40.142024 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.141990 6343 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:40:40.142093 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.142037 6343 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:40:40.142093 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.142058 6343 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:40:40.142193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.142099 6343 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:40:40.142022503Z","UUID":"20910c3d-82ce-4adc-87a5-b8e518aeaa20","Handler":null,"Name":"","Endpoint":""} Nov 02 17:40:40.143364 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.143338 6343 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:40:40.143364 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.143361 6343 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:40:40.143651 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:40.143633 6343 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:40:40.241955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.241894 6343 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:40:40.244929 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.244910 6343 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:40:40.245020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.244947 6343 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:40:40.245020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.244963 6343 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:40:40.245116 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.245022 6343 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.257153 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.257136 6343 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.257203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.257194 6343 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.312581 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.312540 6343 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal"] Nov 02 17:40:40.313204 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313190 6343 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.313285 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313239 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"07e33052aa2d73769dbf3107e42483d7","Type":"ContainerStarted","Data":"36b98675395c780970a62008e4d520290df416f97c74b3e6ec6a6752e3cd5f50"} Nov 02 17:40:40.313414 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313292 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"07e33052aa2d73769dbf3107e42483d7","Type":"ContainerStarted","Data":"57b0cbce8abcb6d3c61b08d17e36827b2e4e719b334830e32ce5ebd2f74f248b"} Nov 02 17:40:40.313414 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313342 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"3f4115a2f4c3236862e1ddc68e0eda82","Type":"ContainerStarted","Data":"ffdac605b8dc8e0ea5e61d8861ae3a2dbed6feb23eb3c6ab4266f7e640703238"} Nov 02 17:40:40.313414 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313363 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"3f4115a2f4c3236862e1ddc68e0eda82","Type":"ContainerDied","Data":"27b70c98a073f1bb30277b3b1bdee1a016de81c46af55f5dc7e8346a1e39e36a"} Nov 02 17:40:40.313414 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313379 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" event={"ID":"3f4115a2f4c3236862e1ddc68e0eda82","Type":"ContainerStarted","Data":"7dd798ae0eeae53c250ff0edf012f41990b0825f91d7dccf8d32ac9b1469390d"} Nov 02 17:40:40.313536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.313506 6343 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.327037 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.327020 6343 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:40.327108 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:40.327069 6343 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.327747 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.327732 6343 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:40.327792 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:40.327759 6343 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.405139 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.405104 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07e33052aa2d73769dbf3107e42483d7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"07e33052aa2d73769dbf3107e42483d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.405139 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.405139 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.405294 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.405162 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.505772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.505685 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07e33052aa2d73769dbf3107e42483d7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"07e33052aa2d73769dbf3107e42483d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.505772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.505729 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.505772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.505757 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.505983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.505775 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07e33052aa2d73769dbf3107e42483d7-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"07e33052aa2d73769dbf3107e42483d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.505983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.505832 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.505983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.505810 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f4115a2f4c3236862e1ddc68e0eda82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" (UID: \"3f4115a2f4c3236862e1ddc68e0eda82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:40.961265 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.961235 6343 apiserver.go:52] "Watching apiserver" Nov 02 17:40:40.979227 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.979202 6343 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:40:40.979824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.979802 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zw4rm","openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5","openshift-ovn-kubernetes/ovnkube-node-nvj7s","openshift-ingress-canary/ingress-canary-88nck","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6","openshift-network-diagnostics/network-check-target-89pdp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd","openshift-multus/multus-hqxnx","openshift-multus/network-metrics-daemon-rhqwb","kube-system/global-pull-secret-syncer-xmv2b","openshift-dns/dns-default-mgw6g","openshift-dns/node-resolver-zb589","openshift-insights/insights-runtime-extractor-kk5x9","openshift-monitoring/monitoring-plugin-77f4f76594-dls4w","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal","openshift-image-registry/node-ca-25nk6","openshift-monitoring/node-exporter-7jkqt","openshift-monitoring/telemeter-client-ffdb45754-7bx92","openshift-console/console-6b5f44b9cf-n4s6q","openshift-network-operator/iptables-alerter-246zz","kube-system/konnectivity-agent-rkl7h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97","openshift-cluster-node-tuning-operator/tuned-dbxfx","openshift-monitoring/prometheus-k8s-0"] Nov 02 17:40:40.984956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.984936 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b2827\"" Nov 02 17:40:40.985067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.984975 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zvsl\"" Nov 02 17:40:40.987798 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.987762 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:40:40.988105 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.988082 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.988197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.988157 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.988431 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.988409 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:40:40.988576 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.988557 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jk79z\"" Nov 02 17:40:40.988815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.988796 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:40:40.989251 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.989232 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:40:40.989665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.989647 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:40:40.989743 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.989723 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.989916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.989889 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjwn5\"" Nov 02 17:40:40.989981 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.989959 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.990033 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.989723 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:40:40.990113 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.990075 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4dl6t\"" Nov 02 17:40:40.990169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.990123 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.990169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.990159 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:40:40.990289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.990192 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.990289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.990250 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.990416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.990337 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.991082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991063 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.991191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991132 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:40:40.991314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991293 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.991402 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991302 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:40:40.991402 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991385 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:40:40.991402 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991397 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6x7d8\"" Nov 02 17:40:40.991582 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991443 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:40:40.991582 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991504 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.991582 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991565 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:40:40.991885 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.991863 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.992189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992168 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.992662 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992642 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kkqh6\"" Nov 02 17:40:40.992740 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992695 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:40:40.992740 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992700 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mkftb\"" Nov 02 17:40:40.992740 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992703 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mg2x\"" Nov 02 17:40:40.992878 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992828 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:40:40.992926 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992898 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpgl9\"" Nov 02 17:40:40.992973 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992946 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs2ss\"" Nov 02 17:40:40.993040 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.993022 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:40:40.993095 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.992651 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zqcnj\"" Nov 02 17:40:40.993095 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.993060 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzjcr\"" Nov 02 17:40:40.993171 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.993107 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7flsb\"" Nov 02 17:40:40.994168 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994149 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:40:40.994291 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994266 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.994356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994346 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.994410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994366 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:40:40.994410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994368 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 17:40:40.994410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994392 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.994560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994412 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:40:40.994560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994456 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.994560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994527 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:40:40.994560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.994557 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:40:40.998758 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.998734 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:40:40.998986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.998971 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 17:40:40.999067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.999001 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 17:40:40.999137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.999095 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 17:40:40.999137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.999101 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ntx78\"" Nov 02 17:40:40.999137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.999124 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 17:40:40.999137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:40.999124 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:40:41.000309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000288 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 17:40:41.000404 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000329 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:40:41.000404 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000365 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:40:41.000404 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000394 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:40:41.000404 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000405 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:40:41.000538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000399 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gjv4t\"" Nov 02 17:40:41.000538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000421 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:40:41.000538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.000477 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-84l5b\"" Nov 02 17:40:41.001836 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001822 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:40:41.001836 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001824 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:40:41.001933 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001824 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:40:41.001933 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001830 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:40:41.001933 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001833 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:40:41.001933 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001903 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5qkwb\"" Nov 02 17:40:41.002061 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001949 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:40:41.002061 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.001992 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:40:41.002061 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.002025 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wbwjw\"" Nov 02 17:40:41.003792 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.003775 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 17:40:41.006924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.006906 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-tmp-dir\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.006972 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.006947 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-etc-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.007005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.006973 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-registration-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.007034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.006999 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwnb\" (UniqueName: \"kubernetes.io/projected/094f3261-e65e-4fd0-893d-0e9c0f99988f-kube-api-access-pxwnb\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.007086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007028 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-systemd-units\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.007086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007070 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:40:41.007167 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007121 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7f4\" (UniqueName: \"kubernetes.io/projected/12700290-2871-451f-9f86-3ea2830de15d-kube-api-access-2p7f4\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.007167 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007141 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-tmp-dir\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.007223 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007181 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-k8s-cni-cncf-io\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.007251 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007232 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r8b\" (UniqueName: \"kubernetes.io/projected/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-kube-api-access-92r8b\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.007326 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007312 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.007425 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007410 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72cc37d8-2509-4902-8494-415871c9db22-data-volume\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.007519 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007502 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-conf-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.007568 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007534 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-socket-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.007568 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007551 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.007568 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007550 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72cc37d8-2509-4902-8494-415871c9db22-data-volume\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.007691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007568 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.007691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007590 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.007691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007625 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.007842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007727 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-kubelet\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.007842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007750 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-socket-dir-parent\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.007842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007765 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-kubernetes\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.007842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007791 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-netns\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.007842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007815 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee56fd64-8479-45f4-b098-16208b7ced9f-hosts-file\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.007842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007833 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6q6\" (UniqueName: \"kubernetes.io/projected/72cc37d8-2509-4902-8494-415871c9db22-kube-api-access-kn6q6\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007871 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20c5900d-985b-4db3-8232-5c24ff85cc01-agent-certs\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007876 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007907 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.007918 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008018 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-slash\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008105 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-config\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008145 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee56fd64-8479-45f4-b098-16208b7ced9f-tmp-dir\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.008154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008144 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20c5900d-985b-4db3-8232-5c24ff85cc01-agent-certs\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008223 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee56fd64-8479-45f4-b098-16208b7ced9f-tmp-dir\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008235 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20c5900d-985b-4db3-8232-5c24ff85cc01-konnectivity-ca\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008263 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-daemon-config\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008290 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008305 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-original-pull-secret\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008331 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-config\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008345 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008378 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-metrics-tls\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008462 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20c5900d-985b-4db3-8232-5c24ff85cc01-konnectivity-ca\") pod \"konnectivity-agent-rkl7h\" (UID: \"20c5900d-985b-4db3-8232-5c24ff85cc01\") " pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:40:41.008500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008474 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-var-lib-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008526 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-metrics-tls\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008539 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e998028-a165-4d2b-8c14-c422d8a4d099-metrics-certs\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008547 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008560 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-daemon-config\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008564 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12700290-2871-451f-9f86-3ea2830de15d-host\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008605 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-bin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008630 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68c1bf64-b418-49c2-9753-4fad4bff08ae-host-slash\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008645 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-sys\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008667 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-dbus\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008701 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008745 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-bin\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008767 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pr69\" (UniqueName: \"kubernetes.io/projected/98c18ffc-40f4-4360-9095-b08f95cab47c-kube-api-access-9pr69\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008786 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12700290-2871-451f-9f86-3ea2830de15d-serviceca\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.008824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008818 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008844 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-kubelet\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008878 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68c1bf64-b418-49c2-9753-4fad4bff08ae-iptables-alerter-script\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008979 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.008989 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysconfig\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009023 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-var-lib-kubelet\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009042 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68c1bf64-b418-49c2-9753-4fad4bff08ae-iptables-alerter-script\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009085 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72cc37d8-2509-4902-8494-415871c9db22-crio-socket\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009104 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-ovn\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009119 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-log-socket\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009134 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-netd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009151 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflxg\" (UniqueName: \"kubernetes.io/projected/ee56fd64-8479-45f4-b098-16208b7ced9f-kube-api-access-zflxg\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009167 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72cc37d8-2509-4902-8494-415871c9db22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009190 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-system-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009216 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-config-volume\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009239 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98c18ffc-40f4-4360-9095-b08f95cab47c-ovn-node-metrics-cert\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009261 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cni-binary-copy\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009292 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-multus\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009319 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72cc37d8-2509-4902-8494-415871c9db22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009321 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-device-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009344 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009375 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4n6r\" (UniqueName: \"kubernetes.io/projected/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kube-api-access-b4n6r\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009361 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12700290-2871-451f-9f86-3ea2830de15d-serviceca\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009455 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-config-volume\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009466 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-host\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009509 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-tmp\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009513 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cni-binary-copy\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009526 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98c18ffc-40f4-4360-9095-b08f95cab47c-ovn-node-metrics-cert\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009534 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009560 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009564 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-tmp\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009630 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009704 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009743 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009772 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009766 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009824 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009824 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009819 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmqr\" (UniqueName: \"kubernetes.io/projected/8271b561-5608-4f0c-93eb-0850dcdcd9c6-kube-api-access-gqmqr\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009855 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009871 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdgd\" (UniqueName: \"kubernetes.io/projected/9e998028-a165-4d2b-8c14-c422d8a4d099-kube-api-access-lkdgd\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009890 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40498a67-6c10-479b-b130-3c97cdd76774-cert\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009906 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-sys-fs\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009927 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-modprobe-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.009996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.009948 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010040 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-run\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010084 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-lib-modules\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010102 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72cc37d8-2509-4902-8494-415871c9db22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010121 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-multus-certs\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010159 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40498a67-6c10-479b-b130-3c97cdd76774-cert\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010237 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-systemd\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010291 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010314 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010686 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72cc37d8-2509-4902-8494-415871c9db22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010701 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4fr\" (UniqueName: \"kubernetes.io/projected/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-kube-api-access-jj4fr\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010730 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010748 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010763 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-script-lib\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010779 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cnibin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010793 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-netns\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010808 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-kubelet-config\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.011158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010841 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-cnibin\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.010888 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011219 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-systemd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011272 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-hostroot\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011342 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-etc-kubernetes\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011407 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnjw\" (UniqueName: \"kubernetes.io/projected/40498a67-6c10-479b-b130-3c97cdd76774-kube-api-access-xvnjw\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011409 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011458 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-ovnkube-script-lib\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011691 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6zm\" (UniqueName: \"kubernetes.io/projected/68c1bf64-b418-49c2-9753-4fad4bff08ae-kube-api-access-2f6zm\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.011862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011741 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:40:41.012272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011881 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-conf\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.012272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.011899 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094f3261-e65e-4fd0-893d-0e9c0f99988f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.012272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012116 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-tuned\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.012272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012155 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.012272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012250 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-node-log\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.012272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012256 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:40:41.012522 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012309 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-tuned\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.012522 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012441 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.012607 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012527 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.012607 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012566 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-env-overrides\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.012766 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012747 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98c18ffc-40f4-4360-9095-b08f95cab47c-env-overrides\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.012899 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.012879 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.013942 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.013919 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:40:41.014022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014008 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:40:41.014661 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014641 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bb3vd27e74ksj\"" Nov 02 17:40:41.014736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014705 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:40:41.014801 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014782 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 17:40:41.014851 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014813 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:40:41.014912 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014895 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:40:41.014960 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.014927 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:40:41.015030 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015015 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5pnjw\"" Nov 02 17:40:41.015141 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015125 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:40:41.015197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015179 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:40:41.015258 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015184 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 17:40:41.015258 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015208 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:40:41.015258 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015249 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:40:41.015389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015259 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:40:41.015389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015268 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:40:41.015389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015284 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 17:40:41.015389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015312 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 17:40:41.015389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.015317 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:40:41.016523 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016502 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zmf58\"" Nov 02 17:40:41.016608 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016581 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qqmv6\"" Nov 02 17:40:41.016608 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016592 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:40:41.016673 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016594 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:40:41.016673 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016618 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 17:40:41.016673 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016637 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:40:41.016769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016701 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 17:40:41.016769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016724 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zwhs4\"" Nov 02 17:40:41.016769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016732 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lg2nk\"" Nov 02 17:40:41.016849 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.016777 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:40:41.017190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.017178 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:40:41.017952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.017938 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:40:41.018302 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.018285 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:40:41.019579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.019565 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:40:41.022891 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.022875 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflxg\" (UniqueName: \"kubernetes.io/projected/ee56fd64-8479-45f4-b098-16208b7ced9f-kube-api-access-zflxg\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.023552 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.023524 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwnb\" (UniqueName: \"kubernetes.io/projected/094f3261-e65e-4fd0-893d-0e9c0f99988f-kube-api-access-pxwnb\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.023713 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.023693 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6q6\" (UniqueName: \"kubernetes.io/projected/72cc37d8-2509-4902-8494-415871c9db22-kube-api-access-kn6q6\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.023841 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.023822 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r8b\" (UniqueName: \"kubernetes.io/projected/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-kube-api-access-92r8b\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.024005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.023990 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7f4\" (UniqueName: \"kubernetes.io/projected/12700290-2871-451f-9f86-3ea2830de15d-kube-api-access-2p7f4\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.024108 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.024094 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pr69\" (UniqueName: \"kubernetes.io/projected/98c18ffc-40f4-4360-9095-b08f95cab47c-kube-api-access-9pr69\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.027879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.027860 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4n6r\" (UniqueName: \"kubernetes.io/projected/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kube-api-access-b4n6r\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.031133 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.031107 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmqr\" (UniqueName: \"kubernetes.io/projected/8271b561-5608-4f0c-93eb-0850dcdcd9c6-kube-api-access-gqmqr\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.031295 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.031274 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnjw\" (UniqueName: \"kubernetes.io/projected/40498a67-6c10-479b-b130-3c97cdd76774-kube-api-access-xvnjw\") pod \"ingress-canary-88nck\" (UID: \"40498a67-6c10-479b-b130-3c97cdd76774\") " pod="openshift-ingress-canary/ingress-canary-88nck" Nov 02 17:40:41.031429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.031404 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") pod \"console-6b5f44b9cf-n4s6q\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:41.031529 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.031409 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdgd\" (UniqueName: \"kubernetes.io/projected/9e998028-a165-4d2b-8c14-c422d8a4d099-kube-api-access-lkdgd\") pod \"network-metrics-daemon-rhqwb\" (UID: \"9e998028-a165-4d2b-8c14-c422d8a4d099\") " pod="openshift-multus/network-metrics-daemon-rhqwb" Nov 02 17:40:41.031891 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.031867 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6zm\" (UniqueName: \"kubernetes.io/projected/68c1bf64-b418-49c2-9753-4fad4bff08ae-kube-api-access-2f6zm\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.032357 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.032331 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4fr\" (UniqueName: \"kubernetes.io/projected/6440b09a-ee4a-4ffa-8102-da0d1d3e10bb-kube-api-access-jj4fr\") pod \"dns-default-mgw6g\" (UID: \"6440b09a-ee4a-4ffa-8102-da0d1d3e10bb\") " pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:41.032731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.032715 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sl6f\" (UniqueName: \"kubernetes.io/projected/3a654875-bb1c-4b4e-834d-9e8001f2a351-kube-api-access-8sl6f\") pod \"network-check-target-89pdp\" (UID: \"3a654875-bb1c-4b4e-834d-9e8001f2a351\") " pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:41.082638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.082615 6343 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:41.082760 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.082742 6343 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:41.091566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.091548 6343 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:40:41.094587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.094569 6343 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:41.094650 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:41.094616 6343 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:41.097186 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.097109 6343 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:41.097373 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:41.097204 6343 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-east-2.compute.internal" Nov 02 17:40:41.113189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113165 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-conf-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.113268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113190 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-root\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.113268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113208 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-serving-certs-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.113268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113223 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.113268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113238 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45684\" (UniqueName: \"kubernetes.io/projected/4a7e0208-a8bc-43f2-8921-5f3d0909120a-kube-api-access-45684\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.113456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113362 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.113456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113376 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-conf-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.113456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113399 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.113456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113427 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-kubernetes\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.113456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113428 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-serving-certs-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.113456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113453 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113501 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-netns\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113536 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee56fd64-8479-45f4-b098-16208b7ced9f-hosts-file\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113561 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbdb\" (UniqueName: \"kubernetes.io/projected/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-api-access-lpbdb\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113577 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-kubernetes\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113592 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-netns\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113582 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113627 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-federate-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113643 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee56fd64-8479-45f4-b098-16208b7ced9f-hosts-file\") pod \"node-resolver-zb589\" (UID: \"ee56fd64-8479-45f4-b098-16208b7ced9f\") " pod="openshift-dns/node-resolver-zb589" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113654 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-slash\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113659 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113684 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113695 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113710 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.113736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113735 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-var-lib-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113759 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-slash\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113738 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113762 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113799 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113799 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12700290-2871-451f-9f86-3ea2830de15d-host\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113834 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12700290-2871-451f-9f86-3ea2830de15d-host\") pod \"node-ca-25nk6\" (UID: \"12700290-2871-451f-9f86-3ea2830de15d\") " pod="openshift-image-registry/node-ca-25nk6" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113835 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68c1bf64-b418-49c2-9753-4fad4bff08ae-host-slash\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113849 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113868 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-sys\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113870 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68c1bf64-b418-49c2-9753-4fad4bff08ae-host-slash\") pod \"iptables-alerter-246zz\" (UID: \"68c1bf64-b418-49c2-9753-4fad4bff08ae\") " pod="openshift-network-operator/iptables-alerter-246zz" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113874 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-federate-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113904 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-dbus\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113928 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-sys\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113937 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysconfig\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113954 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113963 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-var-lib-kubelet\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113975 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysconfig\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113938 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113988 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113904 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-var-lib-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114009 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-dbus\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114027 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-var-lib-kubelet\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.113989 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114088 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114116 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114140 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114165 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114189 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72cc37d8-2509-4902-8494-415871c9db22-crio-socket\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114212 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-ovn\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114236 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-log-socket\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114284 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-log-socket\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114286 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72cc37d8-2509-4902-8494-415871c9db22-crio-socket\") pod \"insights-runtime-extractor-kk5x9\" (UID: \"72cc37d8-2509-4902-8494-415871c9db22\") " pod="openshift-insights/insights-runtime-extractor-kk5x9" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114281 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114357 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-netd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114324 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-netd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114320 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114395 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-ovn\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114389 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114394 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqvg\" (UniqueName: \"kubernetes.io/projected/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-kube-api-access-sbqvg\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114444 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114462 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.114460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114492 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-device-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114495 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114521 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114547 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-device-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114548 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114601 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114618 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114647 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114662 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114665 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114681 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-modprobe-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114711 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-run\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114735 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-lib-modules\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114760 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b1e410-3fff-449a-a344-8ca439c0c9af-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114769 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-modprobe-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114779 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114786 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbml9\" (UniqueName: \"kubernetes.io/projected/bd8dc667-bb79-4137-845c-4cff317422ed-kube-api-access-mbml9\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114787 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114811 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-systemd\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114832 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-lib-modules\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114840 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-run\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114854 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114894 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114906 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114924 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-systemd\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114936 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114960 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114973 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b1e410-3fff-449a-a344-8ca439c0c9af-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.114986 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cnibin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115011 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-netns\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115024 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115037 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-kubelet-config\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115080 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-cnibin\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115090 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-netns\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115103 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115104 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-systemd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115128 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-cnibin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115131 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-hostroot\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115150 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-cnibin\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115153 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd74c07-03e4-44b3-b5d3-4f06a5acb51c-kubelet-config\") pod \"global-pull-secret-syncer-xmv2b\" (UID: \"8bd74c07-03e4-44b3-b5d3-4f06a5acb51c\") " pod="kube-system/global-pull-secret-syncer-xmv2b" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115170 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-etc-kubernetes\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115177 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115191 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-textfile\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115200 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-etc-kubernetes\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115171 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-hostroot\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115178 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-run-systemd\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115217 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115247 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-textfile\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115266 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-node-log\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115288 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115302 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-node-log\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115304 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115328 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115336 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115353 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-etc-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115367 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-registration-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115383 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115393 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-etc-openvswitch\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115397 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-systemd-units\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115403 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115366 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115420 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-systemd-units\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115428 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115463 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115461 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115461 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-registration-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115531 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115566 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-socket-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115584 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115621 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115655 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115671 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-socket-dir\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115678 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115692 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115710 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115730 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd8dc667-bb79-4137-845c-4cff317422ed-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115745 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-kubelet\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115760 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-socket-dir-parent\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115801 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-kubelet\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115826 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115849 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-socket-dir-parent\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115909 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115960 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.115997 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-metrics-client-ca\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116014 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd8dc667-bb79-4137-845c-4cff317422ed-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116022 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116085 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116113 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116129 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116136 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116144 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116186 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-bin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116219 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-wtmp\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116224 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e0208-a8bc-43f2-8921-5f3d0909120a-metrics-client-ca\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116245 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-bin\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116221 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116256 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-bin\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116318 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116326 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98c18ffc-40f4-4360-9095-b08f95cab47c-host-cni-bin\") pod \"ovnkube-node-nvj7s\" (UID: \"98c18ffc-40f4-4360-9095-b08f95cab47c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116328 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116374 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-multus-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116421 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-kubelet\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116435 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116449 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-system-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116464 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-kubelet\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116480 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.116428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116495 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-multus\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116521 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-var-lib-cni-multus\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116527 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-system-cni-dir\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116533 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116554 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-host\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116567 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116572 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116590 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116599 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-host\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116615 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-tls\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116777 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116805 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/31b1e410-3fff-449a-a344-8ca439c0c9af-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116814 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-tls\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116953 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.116751 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-d\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117223 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117239 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxvz\" (UniqueName: \"kubernetes.io/projected/a5a5d720-67fb-459f-823f-ad9993dec96c-kube-api-access-2fxvz\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117255 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117269 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117286 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117312 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117353 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094f3261-e65e-4fd0-893d-0e9c0f99988f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw4rm\" (UID: \"094f3261-e65e-4fd0-893d-0e9c0f99988f\") " pod="openshift-multus/multus-additional-cni-plugins-zw4rm" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117376 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-sys-fs\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117421 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117472 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-sys\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117506 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117514 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-telemeter-client-tls\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117532 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7e0208-a8bc-43f2-8921-5f3d0909120a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117553 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117626 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1-sys-fs\") pod \"aws-ebs-csi-driver-node-f8l97\" (UID: \"9ed1ac36-08cd-4ba5-9f21-74d8616dbdb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8l97" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117694 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a5d720-67fb-459f-823f-ad9993dec96c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117506 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117746 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-multus-certs\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117754 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117765 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117783 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pskb\" (UniqueName: \"kubernetes.io/projected/31b1e410-3fff-449a-a344-8ca439c0c9af-kube-api-access-4pskb\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117792 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-multus-certs\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117805 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117902 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-conf\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117933 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-metrics-client-ca\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117959 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117971 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117971 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.117989 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118005 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118022 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118062 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-k8s-cni-cncf-io\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118090 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5d720-67fb-459f-823f-ad9993dec96c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118109 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118118 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c-etc-sysctl-conf\") pod \"tuned-dbxfx\" (UID: \"12e2028c-d5d1-4af0-95cd-cfcfb8a46d0c\") " pod="openshift-cluster-node-tuning-operator/tuned-dbxfx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118136 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-metrics-client-ca\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118147 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118169 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8271b561-5608-4f0c-93eb-0850dcdcd9c6-host-run-k8s-cni-cncf-io\") pod \"multus-hqxnx\" (UID: \"8271b561-5608-4f0c-93eb-0850dcdcd9c6\") " pod="openshift-multus/multus-hqxnx" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118256 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118273 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a5d720-67fb-459f-823f-ad9993dec96c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.118328 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.120369 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbdb\" (UniqueName: \"kubernetes.io/projected/5701f7c4-97ba-4887-b9fe-4e3bc9941acb-kube-api-access-lpbdb\") pod \"kube-state-metrics-5dc8d7d89b-fbmsd\" (UID: \"5701f7c4-97ba-4887-b9fe-4e3bc9941acb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fbmsd" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.120461 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.120480 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqvg\" (UniqueName: \"kubernetes.io/projected/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-kube-api-access-sbqvg\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.121107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.120588 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45684\" (UniqueName: \"kubernetes.io/projected/4a7e0208-a8bc-43f2-8921-5f3d0909120a-kube-api-access-45684\") pod \"telemeter-client-ffdb45754-7bx92\" (UID: \"4a7e0208-a8bc-43f2-8921-5f3d0909120a\") " pod="openshift-monitoring/telemeter-client-ffdb45754-7bx92" Nov 02 17:40:41.124806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.124789 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbml9\" (UniqueName: \"kubernetes.io/projected/bd8dc667-bb79-4137-845c-4cff317422ed-kube-api-access-mbml9\") pod \"monitoring-plugin-77f4f76594-dls4w\" (UID: \"bd8dc667-bb79-4137-845c-4cff317422ed\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:41.126504 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.126489 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") pod \"alertmanager-main-0\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:40:41.127222 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.127202 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pskb\" (UniqueName: \"kubernetes.io/projected/31b1e410-3fff-449a-a344-8ca439c0c9af-kube-api-access-4pskb\") pod \"prometheus-operator-9cd89b74c-n2gp6\" (UID: \"31b1e410-3fff-449a-a344-8ca439c0c9af\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-n2gp6" Nov 02 17:40:41.127674 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.127655 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxvz\" (UniqueName: \"kubernetes.io/projected/a5a5d720-67fb-459f-823f-ad9993dec96c-kube-api-access-2fxvz\") pod \"openshift-state-metrics-7bd7489957-q5hw5\" (UID: \"a5a5d720-67fb-459f-823f-ad9993dec96c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q5hw5" Nov 02 17:40:41.128385 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.128367 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") pod \"prometheus-k8s-0\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:41.219085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.219013 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-wtmp\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.219085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.219075 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-sys\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.219241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.219111 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-root\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.219241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.219147 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-node-exporter-wtmp\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.219241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.219154 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-sys\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.219241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.219159 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aec52e3c-2a94-48a7-b568-ab54be3ccb9e-root\") pod \"node-exporter-7jkqt\" (UID: \"aec52e3c-2a94-48a7-b568-ab54be3ccb9e\") " pod="openshift-monitoring/node-exporter-7jkqt" Nov 02 17:40:41.299470 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.299451 6343 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:40:41.320779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.320756 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:41.832190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:41.832160 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgw6g" Nov 02 17:40:42.338153 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:42.338120 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:42.342717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:42.342691 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dls4w" Nov 02 17:40:43.312354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:43.312295 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:43.312595 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:43.312577 6343 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:43.708985 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:43.708962 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rkl7h" Nov 02 17:40:44.096488 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:44.096457 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"b43b1f44357e3e8cde5b2a5a1e5ee9db6a4554dd16cb55f990528ad1a79a2349"} Nov 02 17:40:44.096618 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:44.096521 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"84fe8edb2ccaaeec3756366043ac9f79411b6c1bf6036a980b6b1fe9c55e7f8a"} Nov 02 17:40:44.096618 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:44.096548 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"f1fc9e16b55904f70b91bab3f8c5f1b17d76794d490d8b06f2923696bd13bacc"} Nov 02 17:40:44.096618 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:44.096572 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerStarted","Data":"c5c06aede80d8ce7a067a471fb129b7372a6e6394bf423a0d44cca0bf082d00d"} Nov 02 17:40:45.990257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:45.990219 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:45.990852 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:45.990488 6343 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:46.005036 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:46.005015 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:40:46.012077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:46.012037 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:40:47.463738 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:47.463705 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nvj7s" Nov 02 17:40:48.561638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:48.561540 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:40:52.038962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:40:52.038933 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5f44b9cf-n4s6q"] Nov 02 17:40:59.244015 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:40:59.243981 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:41:08.563842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:08.563813 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-89pdp" Nov 02 17:41:15.045028 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:15.044990 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:16.841382 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841345 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:41:16.841968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841719 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="alertmanager" containerID="cri-o://030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51" gracePeriod=120 Nov 02 17:41:16.841968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841782 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-metric" containerID="cri-o://06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e" gracePeriod=120 Nov 02 17:41:16.841968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841810 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy" containerID="cri-o://ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821" gracePeriod=120 Nov 02 17:41:16.841968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841791 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-web" containerID="cri-o://29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba" gracePeriod=120 Nov 02 17:41:16.841968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841914 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="prom-label-proxy" containerID="cri-o://5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5" gracePeriod=120 Nov 02 17:41:16.841968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:16.841941 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="config-reloader" containerID="cri-o://79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888" gracePeriod=120 Nov 02 17:41:17.058394 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.058345 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b5f44b9cf-n4s6q" podUID="67a34000-505d-44f9-95fc-2a1f3cb4e813" containerName="console" containerID="cri-o://d67ea39738ef526880089f146d59feb7687cda2f83f2e84190841a04219ac4c6" gracePeriod=15 Nov 02 17:41:17.216733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216702 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5" exitCode=0 Nov 02 17:41:17.216733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216736 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e" exitCode=0 Nov 02 17:41:17.216733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216745 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821" exitCode=0 Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216754 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888" exitCode=0 Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216761 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51" exitCode=0 Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216771 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5"} Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216847 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e"} Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216885 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821"} Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216916 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888"} Nov 02 17:41:17.217001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.216957 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51"} Nov 02 17:41:17.218474 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.218451 6343 generic.go:356] "Generic (PLEG): container finished" podID="67a34000-505d-44f9-95fc-2a1f3cb4e813" containerID="d67ea39738ef526880089f146d59feb7687cda2f83f2e84190841a04219ac4c6" exitCode=0 Nov 02 17:41:17.218614 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.218502 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5f44b9cf-n4s6q" event={"ID":"67a34000-505d-44f9-95fc-2a1f3cb4e813","Type":"ContainerDied","Data":"d67ea39738ef526880089f146d59feb7687cda2f83f2e84190841a04219ac4c6"} Nov 02 17:41:17.301336 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.301312 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:41:17.336639 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336597 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.336639 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336655 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.336887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336681 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.336887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336704 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.336887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336763 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.336887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336797 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.336887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336827 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.337157 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.336894 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") pod \"67a34000-505d-44f9-95fc-2a1f3cb4e813\" (UID: \"67a34000-505d-44f9-95fc-2a1f3cb4e813\") " Nov 02 17:41:17.337924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.337699 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:17.337924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.337875 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:17.337924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.337885 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:17.338195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.337914 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:17.338195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.337998 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:17.339264 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.339242 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "kube-api-access-9xj2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:17.339515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.339493 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:17.339679 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.339651 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert" pod "67a34000-505d-44f9-95fc-2a1f3cb4e813" (UID: "67a34000-505d-44f9-95fc-2a1f3cb4e813"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.437994 6343 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-oauth-config\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438024 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xj2v\" (UniqueName: \"kubernetes.io/projected/67a34000-505d-44f9-95fc-2a1f3cb4e813-kube-api-access-9xj2v\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438035 6343 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-service-ca\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438068 6343 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-serving-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438078 6343 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a34000-505d-44f9-95fc-2a1f3cb4e813-tmp\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438087 6343 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-console-config\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438097 6343 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-oauth-serving-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:17.438424 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:17.438106 6343 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a34000-505d-44f9-95fc-2a1f3cb4e813-trusted-ca-bundle\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.081855 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.081831 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.143858 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.143760 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.143858 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.143834 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.143870 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.143899 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.143936 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.143973 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144006 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144036 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144093 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144140 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144179 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144223 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.144466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.144267 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") pod \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\" (UID: \"f8aaaba8-6e2f-4193-ac60-8f930a419e00\") " Nov 02 17:41:18.145288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.145251 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:18.145647 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.145618 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:18.147068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.147023 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:18.149699 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.149654 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:18.150358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.150328 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "kube-api-access-6lpmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:18.151699 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.151669 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.151919 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.151863 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.152007 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.151951 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.152289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.152256 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.153235 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.153208 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.154332 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.154300 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:18.154675 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.154652 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.159861 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.159840 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config" pod "f8aaaba8-6e2f-4193-ac60-8f930a419e00" (UID: "f8aaaba8-6e2f-4193-ac60-8f930a419e00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:18.225235 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.225208 6343 generic.go:356] "Generic (PLEG): container finished" podID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerID="29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba" exitCode=0 Nov 02 17:41:18.225354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.225275 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba"} Nov 02 17:41:18.225354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.225287 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.225354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.225331 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8aaaba8-6e2f-4193-ac60-8f930a419e00","Type":"ContainerDied","Data":"85248e95a92d245deb83fac8043b1804dafa807fdc9b476b73b951327d43255e"} Nov 02 17:41:18.225497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.225364 6343 scope.go:122] "RemoveContainer" containerID="5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5" Nov 02 17:41:18.226701 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.226680 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5f44b9cf-n4s6q" event={"ID":"67a34000-505d-44f9-95fc-2a1f3cb4e813","Type":"ContainerDied","Data":"56090f1566ebea9368fcd6e0e67e7340d034ffacdae222bff5d01010efde1ee9"} Nov 02 17:41:18.226791 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.226687 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5f44b9cf-n4s6q" Nov 02 17:41:18.233381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.233358 6343 scope.go:122] "RemoveContainer" containerID="06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e" Nov 02 17:41:18.239885 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.239871 6343 scope.go:122] "RemoveContainer" containerID="ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821" Nov 02 17:41:18.244980 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.244960 6343 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-metrics-client-ca\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.244980 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.244981 6343 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-cluster-tls-config\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.244991 6343 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245003 6343 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-out\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245012 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpmm\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-kube-api-access-6lpmm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245022 6343 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-web-config\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245031 6343 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245039 6343 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aaaba8-6e2f-4193-ac60-8f930a419e00-tls-assets\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245106 6343 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245115 6343 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-secret-alertmanager-main-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245360 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245123 6343 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-main-db\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245360 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245132 6343 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8aaaba8-6e2f-4193-ac60-8f930a419e00-config-volume\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.245360 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.245140 6343 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aaaba8-6e2f-4193-ac60-8f930a419e00-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:18.246018 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.246002 6343 scope.go:122] "RemoveContainer" containerID="29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba" Nov 02 17:41:18.248107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.248040 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5f44b9cf-n4s6q"] Nov 02 17:41:18.250416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.250392 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b5f44b9cf-n4s6q"] Nov 02 17:41:18.252865 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.252846 6343 scope.go:122] "RemoveContainer" containerID="79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888" Nov 02 17:41:18.259234 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.259218 6343 scope.go:122] "RemoveContainer" containerID="030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51" Nov 02 17:41:18.266353 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.266324 6343 scope.go:122] "RemoveContainer" containerID="cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed" Nov 02 17:41:18.266681 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.266662 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:41:18.271805 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.271782 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:41:18.273654 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.273638 6343 scope.go:122] "RemoveContainer" containerID="5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5" Nov 02 17:41:18.273904 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.273884 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5\": container with ID starting with 5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5 not found: ID does not exist" containerID="5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5" Nov 02 17:41:18.273962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.273911 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5"} err="failed to get container status \"5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5\": rpc error: code = NotFound desc = could not find container \"5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5\": container with ID starting with 5e930edd16b30ffb8740ec35151627d54710efd85f2f9f6c2b66e8a1faeac6e5 not found: ID does not exist" Nov 02 17:41:18.273962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.273943 6343 scope.go:122] "RemoveContainer" containerID="06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e" Nov 02 17:41:18.274220 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.274204 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e\": container with ID starting with 06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e not found: ID does not exist" containerID="06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e" Nov 02 17:41:18.274277 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274224 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e"} err="failed to get container status \"06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e\": rpc error: code = NotFound desc = could not find container \"06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e\": container with ID starting with 06c6e7a77a6667dc6e741bc798c50ab7852466950cfcf8ec9f891982b3d6977e not found: ID does not exist" Nov 02 17:41:18.274277 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274237 6343 scope.go:122] "RemoveContainer" containerID="ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821" Nov 02 17:41:18.274482 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.274469 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821\": container with ID starting with ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821 not found: ID does not exist" containerID="ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821" Nov 02 17:41:18.274547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274486 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821"} err="failed to get container status \"ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821\": rpc error: code = NotFound desc = could not find container \"ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821\": container with ID starting with ccb974dcb830b5efd24ad1b4b5d86d81435a35b0d64583b467b9ff68e1a5e821 not found: ID does not exist" Nov 02 17:41:18.274547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274496 6343 scope.go:122] "RemoveContainer" containerID="29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba" Nov 02 17:41:18.274720 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.274702 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba\": container with ID starting with 29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba not found: ID does not exist" containerID="29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba" Nov 02 17:41:18.274760 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274726 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba"} err="failed to get container status \"29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba\": rpc error: code = NotFound desc = could not find container \"29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba\": container with ID starting with 29d80d7d8e511fafcc3b644f87747476876c43611534fb4576fcc372919481ba not found: ID does not exist" Nov 02 17:41:18.274760 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274742 6343 scope.go:122] "RemoveContainer" containerID="79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888" Nov 02 17:41:18.274959 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.274945 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888\": container with ID starting with 79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888 not found: ID does not exist" containerID="79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888" Nov 02 17:41:18.274999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274962 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888"} err="failed to get container status \"79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888\": rpc error: code = NotFound desc = could not find container \"79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888\": container with ID starting with 79196e2f90ac0d3c7cc1cd1626b9defef60538ef5c5eb6eb14b43142eb629888 not found: ID does not exist" Nov 02 17:41:18.274999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.274972 6343 scope.go:122] "RemoveContainer" containerID="030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51" Nov 02 17:41:18.275180 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.275165 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51\": container with ID starting with 030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51 not found: ID does not exist" containerID="030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51" Nov 02 17:41:18.275223 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.275183 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51"} err="failed to get container status \"030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51\": rpc error: code = NotFound desc = could not find container \"030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51\": container with ID starting with 030ff533169a582130aa8ca3578aa4cf7f23f2cf7fc5491372b5f19a82f05a51 not found: ID does not exist" Nov 02 17:41:18.275223 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.275195 6343 scope.go:122] "RemoveContainer" containerID="cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed" Nov 02 17:41:18.275373 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:18.275360 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed\": container with ID starting with cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed not found: ID does not exist" containerID="cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed" Nov 02 17:41:18.275409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.275374 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed"} err="failed to get container status \"cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed\": rpc error: code = NotFound desc = could not find container \"cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed\": container with ID starting with cdc7cd6c42d68d5ce234729c232ab037b135866a2690d6e21b21f1303b317fed not found: ID does not exist" Nov 02 17:41:18.275409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.275385 6343 scope.go:122] "RemoveContainer" containerID="d67ea39738ef526880089f146d59feb7687cda2f83f2e84190841a04219ac4c6" Nov 02 17:41:18.312475 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312436 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:41:18.312795 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312779 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-web" Nov 02 17:41:18.312795 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312796 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-web" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312827 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="init-config-reloader" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312833 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="init-config-reloader" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312844 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-metric" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312849 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-metric" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312855 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="prom-label-proxy" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312860 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="prom-label-proxy" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312865 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="config-reloader" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312870 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="config-reloader" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312878 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312882 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312894 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="alertmanager" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312901 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="alertmanager" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312910 6343 cpu_manager.go:414] "containerMap: removing container" podUID="67a34000-505d-44f9-95fc-2a1f3cb4e813" containerName="console" Nov 02 17:41:18.312915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312915 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a34000-505d-44f9-95fc-2a1f3cb4e813" containerName="console" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312953 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="init-config-reloader" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312961 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-metric" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312966 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="prom-label-proxy" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312970 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="config-reloader" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312975 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312980 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="alertmanager" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312985 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a34000-505d-44f9-95fc-2a1f3cb4e813" containerName="console" Nov 02 17:41:18.313330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.312991 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" containerName="kube-rbac-proxy-web" Nov 02 17:41:18.316130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.316108 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.332027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.331969 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:41:18.332027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.331989 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:41:18.332027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332001 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:41:18.332027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332012 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zmf58\"" Nov 02 17:41:18.332027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332006 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:41:18.332027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.331992 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:41:18.332350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332042 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:41:18.332350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332068 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:41:18.332350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332030 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:41:18.332350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.332108 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:41:18.336493 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.336466 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:41:18.346218 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346185 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e765946-a617-4fc7-b66d-7e5bf4273174-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346221 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346244 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e765946-a617-4fc7-b66d-7e5bf4273174-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346260 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346450 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e765946-a617-4fc7-b66d-7e5bf4273174-config-out\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346486 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346544 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47xw\" (UniqueName: \"kubernetes.io/projected/1e765946-a617-4fc7-b66d-7e5bf4273174-kube-api-access-g47xw\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346584 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346773 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346677 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346773 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346708 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-web-config\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346773 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346739 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346773 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346765 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e765946-a617-4fc7-b66d-7e5bf4273174-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.346939 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.346886 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e765946-a617-4fc7-b66d-7e5bf4273174-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447436 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447345 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e765946-a617-4fc7-b66d-7e5bf4273174-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447436 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447392 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447436 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447416 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e765946-a617-4fc7-b66d-7e5bf4273174-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447436 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447433 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447455 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e765946-a617-4fc7-b66d-7e5bf4273174-config-out\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447473 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447490 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47xw\" (UniqueName: \"kubernetes.io/projected/1e765946-a617-4fc7-b66d-7e5bf4273174-kube-api-access-g47xw\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447505 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447523 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447549 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-web-config\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447585 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447622 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e765946-a617-4fc7-b66d-7e5bf4273174-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.447781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447655 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e765946-a617-4fc7-b66d-7e5bf4273174-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.448300 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447836 6343 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:41:18.448300 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.447971 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e765946-a617-4fc7-b66d-7e5bf4273174-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.448468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.448417 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e765946-a617-4fc7-b66d-7e5bf4273174-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.448830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.448805 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e765946-a617-4fc7-b66d-7e5bf4273174-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.450834 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.450805 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.450928 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.450855 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e765946-a617-4fc7-b66d-7e5bf4273174-config-out\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.450928 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.450868 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-web-config\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.451041 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.450992 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.451132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.451068 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.451193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.451157 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.451319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.451301 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.452179 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.452161 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e765946-a617-4fc7-b66d-7e5bf4273174-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.452269 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.452248 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e765946-a617-4fc7-b66d-7e5bf4273174-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.456725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.456709 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47xw\" (UniqueName: \"kubernetes.io/projected/1e765946-a617-4fc7-b66d-7e5bf4273174-kube-api-access-g47xw\") pod \"alertmanager-main-0\" (UID: \"1e765946-a617-4fc7-b66d-7e5bf4273174\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.637457 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.637428 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:41:18.916858 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:18.916834 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:41:18.922571 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:41:18.922505 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e765946_a617_4fc7_b66d_7e5bf4273174.slice/crio-e1af4e30fdcbe6e59531696e908c975dd66aaff426605c84d548da04fa792556 WatchSource:0}: Error finding container e1af4e30fdcbe6e59531696e908c975dd66aaff426605c84d548da04fa792556: Status 404 returned error can't find the container with id e1af4e30fdcbe6e59531696e908c975dd66aaff426605c84d548da04fa792556 Nov 02 17:41:19.233547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:19.233467 6343 generic.go:356] "Generic (PLEG): container finished" podID="1e765946-a617-4fc7-b66d-7e5bf4273174" containerID="94d7fc89d029fdfc67646d49e4bfa8a473d0578370352897535e365e59d0c14a" exitCode=0 Nov 02 17:41:19.233547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:19.233501 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerDied","Data":"94d7fc89d029fdfc67646d49e4bfa8a473d0578370352897535e365e59d0c14a"} Nov 02 17:41:19.233547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:19.233548 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"e1af4e30fdcbe6e59531696e908c975dd66aaff426605c84d548da04fa792556"} Nov 02 17:41:20.022646 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.022610 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a34000-505d-44f9-95fc-2a1f3cb4e813" path="/var/lib/kubelet/pods/67a34000-505d-44f9-95fc-2a1f3cb4e813/volumes" Nov 02 17:41:20.023039 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.023025 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8aaaba8-6e2f-4193-ac60-8f930a419e00" path="/var/lib/kubelet/pods/f8aaaba8-6e2f-4193-ac60-8f930a419e00/volumes" Nov 02 17:41:20.239457 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.239373 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"188a24e3a9e049084163387b3a65d64cf991611dbc5121ed3d8a31f63ee8b6f6"} Nov 02 17:41:20.239457 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.239434 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"72fd1a43447bf71b34f0dbadb7d9ba5a6b0d75bd06502369da45b464b3f5e5ab"} Nov 02 17:41:20.239883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.239465 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"bd3166ee60c0fb9c9f42e562654d640156589fba9f1209e07f6da573e8eaa22b"} Nov 02 17:41:20.239883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.239489 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"bf26d93b3c6cb305b6fc8006d65014a6bc1d26b93ee11ad960a733a85c498d96"} Nov 02 17:41:20.239883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.239513 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"05004d7fd71e5b52b33eb4661033fbd0b40f0e7dc563b00b76246e43108c5036"} Nov 02 17:41:20.239883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.239542 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e765946-a617-4fc7-b66d-7e5bf4273174","Type":"ContainerStarted","Data":"283eec40af393544c45072a5c79249ba8c99d2d47a54f9b6425cab814fe65de7"} Nov 02 17:41:20.273329 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:20.273285 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.260863255 podStartE2EDuration="2.273273195s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.40994ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:20.273008908 +0000 UTC m=+40.372037106" watchObservedRunningTime="2026-11-02 17:41:20.273273195 +0000 UTC m=+40.372301438" Nov 02 17:41:21.108572 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.108521 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:41:21.108970 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.108939 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="prometheus" containerID="cri-o://09d406e6e8aee219d4de6693756da21712b3dd2ba319dda0b5e4ed69209effe1" gracePeriod=600 Nov 02 17:41:21.109227 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.108999 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b43b1f44357e3e8cde5b2a5a1e5ee9db6a4554dd16cb55f990528ad1a79a2349" gracePeriod=600 Nov 02 17:41:21.109320 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.109082 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-web" containerID="cri-o://f1fc9e16b55904f70b91bab3f8c5f1b17d76794d490d8b06f2923696bd13bacc" gracePeriod=600 Nov 02 17:41:21.109392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.109096 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy" containerID="cri-o://84fe8edb2ccaaeec3756366043ac9f79411b6c1bf6036a980b6b1fe9c55e7f8a" gracePeriod=600 Nov 02 17:41:21.109392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.109113 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="config-reloader" containerID="cri-o://72ee5e4a15eda0d55245ac86207d753278932acb1b82debffe4284fc64b205d2" gracePeriod=600 Nov 02 17:41:21.109496 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.109126 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="thanos-sidecar" containerID="cri-o://c5c06aede80d8ce7a067a471fb129b7372a6e6394bf423a0d44cca0bf082d00d" gracePeriod=600 Nov 02 17:41:21.247263 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247238 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="b43b1f44357e3e8cde5b2a5a1e5ee9db6a4554dd16cb55f990528ad1a79a2349" exitCode=0 Nov 02 17:41:21.247263 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247262 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="84fe8edb2ccaaeec3756366043ac9f79411b6c1bf6036a980b6b1fe9c55e7f8a" exitCode=0 Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247273 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="f1fc9e16b55904f70b91bab3f8c5f1b17d76794d490d8b06f2923696bd13bacc" exitCode=0 Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247282 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="c5c06aede80d8ce7a067a471fb129b7372a6e6394bf423a0d44cca0bf082d00d" exitCode=0 Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247287 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="72ee5e4a15eda0d55245ac86207d753278932acb1b82debffe4284fc64b205d2" exitCode=0 Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247294 6343 generic.go:356] "Generic (PLEG): container finished" podID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerID="09d406e6e8aee219d4de6693756da21712b3dd2ba319dda0b5e4ed69209effe1" exitCode=0 Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247314 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"b43b1f44357e3e8cde5b2a5a1e5ee9db6a4554dd16cb55f990528ad1a79a2349"} Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247406 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"84fe8edb2ccaaeec3756366043ac9f79411b6c1bf6036a980b6b1fe9c55e7f8a"} Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247448 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"f1fc9e16b55904f70b91bab3f8c5f1b17d76794d490d8b06f2923696bd13bacc"} Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247488 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"c5c06aede80d8ce7a067a471fb129b7372a6e6394bf423a0d44cca0bf082d00d"} Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247528 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"72ee5e4a15eda0d55245ac86207d753278932acb1b82debffe4284fc64b205d2"} Nov 02 17:41:21.247665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.247568 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"09d406e6e8aee219d4de6693756da21712b3dd2ba319dda0b5e4ed69209effe1"} Nov 02 17:41:21.378909 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.378886 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:21.472328 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472281 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472328 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472340 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472361 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472389 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472499 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472561 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472605 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472637 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472669 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472696 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472740 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.472781 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472782 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472779 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472808 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472828 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472833 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472886 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472896 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.472929 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473017 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473043 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473091 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473114 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") pod \"65bcced4-e7fa-44db-ba68-001da2189ec6\" (UID: \"65bcced4-e7fa-44db-ba68-001da2189ec6\") " Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473197 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473227 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473473 6343 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473496 6343 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473513 6343 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473529 6343 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-configmap-metrics-client-ca\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.473585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473545 6343 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.473973 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.473854 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.475219 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.474600 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:21.475219 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.474888 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:21.476079 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.476019 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.476665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.476629 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.477278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477222 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.477442 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477399 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.477614 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477589 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "kube-api-access-47lmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:21.477687 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477600 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:21.477815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477792 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.477915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477854 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.477970 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.477924 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.478109 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.478091 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:21.478500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.478483 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.486203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.486182 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config" pod "65bcced4-e7fa-44db-ba68-001da2189ec6" (UID: "65bcced4-e7fa-44db-ba68-001da2189ec6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:21.574282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574247 6343 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574276 6343 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-db\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574287 6343 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-config\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574297 6343 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-metrics-client-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574307 6343 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574316 6343 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-kube-rbac-proxy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574325 6343 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-web-config\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574333 6343 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574342 6343 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574353 6343 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65bcced4-e7fa-44db-ba68-001da2189ec6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574361 6343 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574370 6343 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65bcced4-e7fa-44db-ba68-001da2189ec6-secret-grpc-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574380 6343 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-tls-assets\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574388 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47lmk\" (UniqueName: \"kubernetes.io/projected/65bcced4-e7fa-44db-ba68-001da2189ec6-kube-api-access-47lmk\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:21.574497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:21.574396 6343 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65bcced4-e7fa-44db-ba68-001da2189ec6-config-out\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:22.253561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.253526 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65bcced4-e7fa-44db-ba68-001da2189ec6","Type":"ContainerDied","Data":"51b046ef081e98dd1df230677be73b8c33c118a8fdbe2cab5a2908b6f90289b0"} Nov 02 17:41:22.254005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.253591 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.254005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.253615 6343 scope.go:122] "RemoveContainer" containerID="b43b1f44357e3e8cde5b2a5a1e5ee9db6a4554dd16cb55f990528ad1a79a2349" Nov 02 17:41:22.261262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.261248 6343 scope.go:122] "RemoveContainer" containerID="84fe8edb2ccaaeec3756366043ac9f79411b6c1bf6036a980b6b1fe9c55e7f8a" Nov 02 17:41:22.268583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.268566 6343 scope.go:122] "RemoveContainer" containerID="f1fc9e16b55904f70b91bab3f8c5f1b17d76794d490d8b06f2923696bd13bacc" Nov 02 17:41:22.274825 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.274811 6343 scope.go:122] "RemoveContainer" containerID="c5c06aede80d8ce7a067a471fb129b7372a6e6394bf423a0d44cca0bf082d00d" Nov 02 17:41:22.277034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.277013 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:41:22.282383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.282360 6343 scope.go:122] "RemoveContainer" containerID="72ee5e4a15eda0d55245ac86207d753278932acb1b82debffe4284fc64b205d2" Nov 02 17:41:22.283330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.283312 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:41:22.288860 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.288844 6343 scope.go:122] "RemoveContainer" containerID="09d406e6e8aee219d4de6693756da21712b3dd2ba319dda0b5e4ed69209effe1" Nov 02 17:41:22.295504 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.295486 6343 scope.go:122] "RemoveContainer" containerID="fb843d2f3022fb6c02b264443328dc82aa8488bf1ba72d318eb26ed758c057b4" Nov 02 17:41:22.309696 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309680 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:41:22.309932 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309921 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-thanos" Nov 02 17:41:22.309974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309934 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-thanos" Nov 02 17:41:22.309974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309951 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="prometheus" Nov 02 17:41:22.309974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309957 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="prometheus" Nov 02 17:41:22.309974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309965 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-web" Nov 02 17:41:22.309974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309970 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-web" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309977 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309982 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309991 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="init-config-reloader" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.309996 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="init-config-reloader" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310010 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="thanos-sidecar" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310015 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="thanos-sidecar" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310025 6343 cpu_manager.go:414] "containerMap: removing container" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="config-reloader" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310029 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="config-reloader" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310116 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="thanos-sidecar" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310128 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="config-reloader" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310134 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-thanos" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310142 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="prometheus" Nov 02 17:41:22.310145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310148 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy-web" Nov 02 17:41:22.310504 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.310153 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" containerName="kube-rbac-proxy" Nov 02 17:41:22.313362 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.313349 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.321002 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.320983 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:41:22.321135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321118 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:41:22.321201 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321134 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:41:22.321267 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321228 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bb3vd27e74ksj\"" Nov 02 17:41:22.321324 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321303 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:41:22.321586 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321567 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:41:22.321586 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321583 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:41:22.321744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321586 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:41:22.321744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321571 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:41:22.321744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321574 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:41:22.321744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321627 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5pnjw\"" Nov 02 17:41:22.321744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321652 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:41:22.321744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.321655 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:41:22.322031 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.322014 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:41:22.322254 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.322242 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:41:22.328527 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.328511 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:41:22.328638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.328621 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:41:22.329973 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.329950 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:41:22.381099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381072 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381108 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381125 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381144 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381260 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381311 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381334 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/860ef26b-655c-4ce5-bfa2-17f312fa81a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381351 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381370 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381423 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381473 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381495 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/860ef26b-655c-4ce5-bfa2-17f312fa81a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381511 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381528 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381544 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381611 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8g5\" (UniqueName: \"kubernetes.io/projected/860ef26b-655c-4ce5-bfa2-17f312fa81a0-kube-api-access-7x8g5\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381629 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381646 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-config\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381659 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.381749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.381675 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482792 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482768 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482799 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482815 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482833 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482850 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482871 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482890 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/860ef26b-655c-4ce5-bfa2-17f312fa81a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.482910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482910 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482927 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482952 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.482981 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483010 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/860ef26b-655c-4ce5-bfa2-17f312fa81a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483032 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483076 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483095 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483121 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8g5\" (UniqueName: \"kubernetes.io/projected/860ef26b-655c-4ce5-bfa2-17f312fa81a0-kube-api-access-7x8g5\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483147 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483175 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-config\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483198 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.483305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483221 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.484094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483668 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.484094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.483706 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.484094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.484080 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.484094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.484086 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.484792 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.484764 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.485950 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.485921 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.486157 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.486135 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/860ef26b-655c-4ce5-bfa2-17f312fa81a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.486592 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.486565 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.486938 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.486914 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.487088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.487064 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.487212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.487155 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.487370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.487319 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.487622 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.487602 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.487688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.487648 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.487752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.487737 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/860ef26b-655c-4ce5-bfa2-17f312fa81a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.488623 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.488601 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.488720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.488675 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-config\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.489204 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.489184 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/860ef26b-655c-4ce5-bfa2-17f312fa81a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.489637 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.489620 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860ef26b-655c-4ce5-bfa2-17f312fa81a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.494513 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.494494 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8g5\" (UniqueName: \"kubernetes.io/projected/860ef26b-655c-4ce5-bfa2-17f312fa81a0-kube-api-access-7x8g5\") pod \"prometheus-k8s-0\" (UID: \"860ef26b-655c-4ce5-bfa2-17f312fa81a0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.630166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.630143 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:22.910848 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:22.910812 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:41:22.912454 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:41:22.912427 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860ef26b_655c_4ce5_bfa2_17f312fa81a0.slice/crio-665014984cb62d115ee1de81b9d11d78cee1241105e58a4aec4e2f0c5c9e638d WatchSource:0}: Error finding container 665014984cb62d115ee1de81b9d11d78cee1241105e58a4aec4e2f0c5c9e638d: Status 404 returned error can't find the container with id 665014984cb62d115ee1de81b9d11d78cee1241105e58a4aec4e2f0c5c9e638d Nov 02 17:41:23.259930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:23.259847 6343 generic.go:356] "Generic (PLEG): container finished" podID="860ef26b-655c-4ce5-bfa2-17f312fa81a0" containerID="0029db3e70dd55bea29d7e8369e168acea50658ce7fc3aafd2f685611409654f" exitCode=0 Nov 02 17:41:23.259930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:23.259889 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerDied","Data":"0029db3e70dd55bea29d7e8369e168acea50658ce7fc3aafd2f685611409654f"} Nov 02 17:41:23.260347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:23.259985 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"665014984cb62d115ee1de81b9d11d78cee1241105e58a4aec4e2f0c5c9e638d"} Nov 02 17:41:24.017621 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.017586 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bcced4-e7fa-44db-ba68-001da2189ec6" path="/var/lib/kubelet/pods/65bcced4-e7fa-44db-ba68-001da2189ec6/volumes" Nov 02 17:41:24.270477 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.270386 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"767ec6208ea53bb1c5439d62978cd5df1be620bbffaefc68ba7800d3877fcf19"} Nov 02 17:41:24.270477 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.270473 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"e0039e2bc2ba53aadf028f251c6f68d0c6e7f1936e4d12677e43b2d362115d78"} Nov 02 17:41:24.270974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.270499 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"eac5bc13b2eda52befe2bcfa3945fac99ee7f612f0ad30cc6f1ee04cf09ff021"} Nov 02 17:41:24.270974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.270524 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"35370c2d674ad432c61133644bf524c1ea9c55ed5c7559ba4d44bf8a0e6fcd23"} Nov 02 17:41:24.270974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.270548 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"365859c00b5dbe2bad91d88ea2e3347be9328cda1ad99786a8873be5747b16f2"} Nov 02 17:41:24.270974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.270571 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"860ef26b-655c-4ce5-bfa2-17f312fa81a0","Type":"ContainerStarted","Data":"f9b70a6999f6269b344d8231b201139881f696762754596e598e5de96657232e"} Nov 02 17:41:24.303535 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:24.303406 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.288784528 podStartE2EDuration="2.303388299s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.603771ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:24.30168506 +0000 UTC m=+44.400713255" watchObservedRunningTime="2026-11-02 17:41:24.303388299 +0000 UTC m=+44.402416493" Nov 02 17:41:27.630691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:41:27.630658 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:41:29.253939 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:29.251920 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:41:59.244654 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:41:59.244612 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:42:10.688129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:42:10.688069 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:42:10.703369 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:42:10.703342 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:42:11.451551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:42:11.451515 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:42:29.244030 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:42:29.243947 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:42:59.245458 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:42:59.245421 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:43:29.243612 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:43:29.243569 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:43:31.009610 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.009576 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6"] Nov 02 17:43:31.012815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.012794 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.021598 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.021581 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 17:43:31.021680 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.021580 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 17:43:31.031182 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.031162 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6"] Nov 02 17:43:31.082673 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.082647 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-54bzl\"" Nov 02 17:43:31.128181 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.128159 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.128289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.128212 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.128377 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.128364 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bxv\" (UniqueName: \"kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.229077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.229031 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.229212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.229097 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.229212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.229126 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bxv\" (UniqueName: \"kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.229409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.229391 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.229447 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.229425 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.238357 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.238331 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bxv\" (UniqueName: \"kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.328094 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.328069 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:31.601602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.601527 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6"] Nov 02 17:43:31.604916 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:43:31.604888 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ebf775_540b_4359_97ff_bbbfdfaa52aa.slice/crio-6049764e581bd0e4feae83fc9c0252177988a7072ea635ebc12ae93331a390b2 WatchSource:0}: Error finding container 6049764e581bd0e4feae83fc9c0252177988a7072ea635ebc12ae93331a390b2: Status 404 returned error can't find the container with id 6049764e581bd0e4feae83fc9c0252177988a7072ea635ebc12ae93331a390b2 Nov 02 17:43:31.716694 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:31.716655 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" event={"ID":"d1ebf775-540b-4359-97ff-bbbfdfaa52aa","Type":"ContainerStarted","Data":"6049764e581bd0e4feae83fc9c0252177988a7072ea635ebc12ae93331a390b2"} Nov 02 17:43:37.741390 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:37.741354 6343 generic.go:356] "Generic (PLEG): container finished" podID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerID="6358264631b3964cbd38967eeba869bbc550124f3f6254d68bcd08dc614acf05" exitCode=0 Nov 02 17:43:37.741793 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:37.741439 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" event={"ID":"d1ebf775-540b-4359-97ff-bbbfdfaa52aa","Type":"ContainerDied","Data":"6358264631b3964cbd38967eeba869bbc550124f3f6254d68bcd08dc614acf05"} Nov 02 17:43:40.754925 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:40.754891 6343 generic.go:356] "Generic (PLEG): container finished" podID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerID="86e13c17070d8f5ba25a3a9fbfa8c07832faa171cd3e1729399df21ce31b81ac" exitCode=0 Nov 02 17:43:40.755357 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:40.754966 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" event={"ID":"d1ebf775-540b-4359-97ff-bbbfdfaa52aa","Type":"ContainerDied","Data":"86e13c17070d8f5ba25a3a9fbfa8c07832faa171cd3e1729399df21ce31b81ac"} Nov 02 17:43:46.781636 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:46.781540 6343 generic.go:356] "Generic (PLEG): container finished" podID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerID="66da9f3d877d038163ebdd10403266679a3105bf1ec18a3f3c9f954020733bc7" exitCode=0 Nov 02 17:43:46.781636 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:46.781592 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" event={"ID":"d1ebf775-540b-4359-97ff-bbbfdfaa52aa","Type":"ContainerDied","Data":"66da9f3d877d038163ebdd10403266679a3105bf1ec18a3f3c9f954020733bc7"} Nov 02 17:43:47.909720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.909694 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:47.967550 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.967520 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle\") pod \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " Nov 02 17:43:47.967696 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.967619 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util\") pod \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " Nov 02 17:43:47.967696 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.967650 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv\" (UniqueName: \"kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv\") pod \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\" (UID: \"d1ebf775-540b-4359-97ff-bbbfdfaa52aa\") " Nov 02 17:43:47.968315 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.968288 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle" pod "d1ebf775-540b-4359-97ff-bbbfdfaa52aa" (UID: "d1ebf775-540b-4359-97ff-bbbfdfaa52aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:47.969713 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.969692 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv" pod "d1ebf775-540b-4359-97ff-bbbfdfaa52aa" (UID: "d1ebf775-540b-4359-97ff-bbbfdfaa52aa"). InnerVolumeSpecName "kube-api-access-n5bxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:47.972349 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:47.972322 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util" pod "d1ebf775-540b-4359-97ff-bbbfdfaa52aa" (UID: "d1ebf775-540b-4359-97ff-bbbfdfaa52aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:48.068885 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:48.068803 6343 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-util\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:48.068885 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:48.068841 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bxv\" (UniqueName: \"kubernetes.io/projected/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-kube-api-access-n5bxv\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:48.068885 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:48.068857 6343 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1ebf775-540b-4359-97ff-bbbfdfaa52aa-bundle\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:48.792214 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:48.792177 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" event={"ID":"d1ebf775-540b-4359-97ff-bbbfdfaa52aa","Type":"ContainerDied","Data":"6049764e581bd0e4feae83fc9c0252177988a7072ea635ebc12ae93331a390b2"} Nov 02 17:43:48.792214 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:48.792224 6343 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6049764e581bd0e4feae83fc9c0252177988a7072ea635ebc12ae93331a390b2" Nov 02 17:43:48.792473 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:48.792230 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czslm6" Nov 02 17:43:54.858992 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.858957 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt"] Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859272 6343 cpu_manager.go:414] "containerMap: removing container" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="extract" Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859284 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="extract" Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859303 6343 cpu_manager.go:414] "containerMap: removing container" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="pull" Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859308 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="pull" Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859327 6343 cpu_manager.go:414] "containerMap: removing container" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="util" Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859333 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="util" Nov 02 17:43:54.859400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.859379 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ebf775-540b-4359-97ff-bbbfdfaa52aa" containerName="extract" Nov 02 17:43:54.862364 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.862347 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:54.881637 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.881614 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:43:54.881734 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.881655 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 17:43:54.893031 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.893014 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-crpj2\"" Nov 02 17:43:54.920009 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.919983 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khzdt\" (UID: \"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:54.920106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.920031 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7hv\" (UniqueName: \"kubernetes.io/projected/3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5-kube-api-access-zs7hv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khzdt\" (UID: \"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:54.945684 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:54.945652 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt"] Nov 02 17:43:55.021368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.021331 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7hv\" (UniqueName: \"kubernetes.io/projected/3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5-kube-api-access-zs7hv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khzdt\" (UID: \"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:55.021537 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.021415 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khzdt\" (UID: \"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:55.021813 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.021793 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khzdt\" (UID: \"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:55.037436 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.037410 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7hv\" (UniqueName: \"kubernetes.io/projected/3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5-kube-api-access-zs7hv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-khzdt\" (UID: \"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:55.173606 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.173528 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" Nov 02 17:43:55.466753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.466673 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt"] Nov 02 17:43:55.470116 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:43:55.470091 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf8911f_bc5c_4c2c_a7ff_d47834cba9d5.slice/crio-b6fff129f4771ea50a29e8dafe6049cca5dfbf43e4c7d78836574c401e66472d WatchSource:0}: Error finding container b6fff129f4771ea50a29e8dafe6049cca5dfbf43e4c7d78836574c401e66472d: Status 404 returned error can't find the container with id b6fff129f4771ea50a29e8dafe6049cca5dfbf43e4c7d78836574c401e66472d Nov 02 17:43:55.821389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:55.821308 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" event={"ID":"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5","Type":"ContainerStarted","Data":"b6fff129f4771ea50a29e8dafe6049cca5dfbf43e4c7d78836574c401e66472d"} Nov 02 17:43:57.831277 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:57.831246 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" event={"ID":"3cf8911f-bc5c-4c2c-a7ff-d47834cba9d5","Type":"ContainerStarted","Data":"c754d7be9cb3f8db4bb967c0819a47ac4fa23f9af6ca2ce0b6197cb6143de501"} Nov 02 17:43:57.878786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:43:57.878736 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-khzdt" podStartSLOduration=2.14788909 podStartE2EDuration="3.878722764s" totalImagesPullingTime="1.730833674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:57.87716693 +0000 UTC m=+197.976195125" watchObservedRunningTime="2026-11-02 17:43:57.878722764 +0000 UTC m=+197.977750959" Nov 02 17:43:59.255928 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:43:59.255884 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:44:01.323183 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.323155 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88"] Nov 02 17:44:01.326573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.326556 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.378812 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.378776 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.378812 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.378813 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.379030 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.378831 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52bf20cf-0da5-496a-ab03-ff53cfc7587f-manifests\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.379030 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.378921 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfq2\" (UniqueName: \"kubernetes.io/projected/52bf20cf-0da5-496a-ab03-ff53cfc7587f-kube-api-access-nhfq2\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.379030 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.378998 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52bf20cf-0da5-496a-ab03-ff53cfc7587f-tmp\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.379222 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.379040 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.379222 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.379200 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:44:01.380573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.380558 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:44:01.381162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.381148 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:44:01.381279 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.381264 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkllc\"" Nov 02 17:44:01.381403 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.381389 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:44:01.382364 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.382352 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:44:01.419922 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.419883 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88"] Nov 02 17:44:01.479631 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.479595 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52bf20cf-0da5-496a-ab03-ff53cfc7587f-tmp\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.479786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.479641 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.479786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.479686 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.479786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.479704 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.479786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.479723 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52bf20cf-0da5-496a-ab03-ff53cfc7587f-manifests\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.479786 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.479742 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfq2\" (UniqueName: \"kubernetes.io/projected/52bf20cf-0da5-496a-ab03-ff53cfc7587f-kube-api-access-nhfq2\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.480201 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.480175 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52bf20cf-0da5-496a-ab03-ff53cfc7587f-manifests\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.482085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.481993 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52bf20cf-0da5-496a-ab03-ff53cfc7587f-tmp\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.482524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.482504 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.482606 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.482565 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.482606 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.482572 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52bf20cf-0da5-496a-ab03-ff53cfc7587f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.502162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.502137 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfq2\" (UniqueName: \"kubernetes.io/projected/52bf20cf-0da5-496a-ab03-ff53cfc7587f-kube-api-access-nhfq2\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88\" (UID: \"52bf20cf-0da5-496a-ab03-ff53cfc7587f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.550848 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.550819 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z"] Nov 02 17:44:01.554297 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.554280 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.564329 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.564304 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 17:44:01.577106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.577019 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9mcfd\"" Nov 02 17:44:01.578074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.578036 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z"] Nov 02 17:44:01.641741 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.641708 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:01.668350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.668322 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 17:44:01.681518 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.681484 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wp9v\" (UniqueName: \"kubernetes.io/projected/80de8a96-1a74-4e82-877a-0cb95a840207-kube-api-access-2wp9v\") pod \"cert-manager-webhook-5dbbbf75cc-dxh8z\" (UID: \"80de8a96-1a74-4e82-877a-0cb95a840207\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.681716 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.681530 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80de8a96-1a74-4e82-877a-0cb95a840207-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dxh8z\" (UID: \"80de8a96-1a74-4e82-877a-0cb95a840207\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.782767 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.782731 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wp9v\" (UniqueName: \"kubernetes.io/projected/80de8a96-1a74-4e82-877a-0cb95a840207-kube-api-access-2wp9v\") pod \"cert-manager-webhook-5dbbbf75cc-dxh8z\" (UID: \"80de8a96-1a74-4e82-877a-0cb95a840207\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.782935 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.782780 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80de8a96-1a74-4e82-877a-0cb95a840207-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dxh8z\" (UID: \"80de8a96-1a74-4e82-877a-0cb95a840207\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.799721 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.799691 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80de8a96-1a74-4e82-877a-0cb95a840207-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dxh8z\" (UID: \"80de8a96-1a74-4e82-877a-0cb95a840207\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.800685 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.800666 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wp9v\" (UniqueName: \"kubernetes.io/projected/80de8a96-1a74-4e82-877a-0cb95a840207-kube-api-access-2wp9v\") pod \"cert-manager-webhook-5dbbbf75cc-dxh8z\" (UID: \"80de8a96-1a74-4e82-877a-0cb95a840207\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:01.876183 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:01.876103 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:02.044087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.044032 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88"] Nov 02 17:44:02.049376 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:02.049353 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bf20cf_0da5_496a_ab03_ff53cfc7587f.slice/crio-7756df67303882fd44bc94e755002c1f69a73787135f96d51d040efc0bc360ed WatchSource:0}: Error finding container 7756df67303882fd44bc94e755002c1f69a73787135f96d51d040efc0bc360ed: Status 404 returned error can't find the container with id 7756df67303882fd44bc94e755002c1f69a73787135f96d51d040efc0bc360ed Nov 02 17:44:02.161378 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.161354 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z"] Nov 02 17:44:02.163358 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:02.163317 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80de8a96_1a74_4e82_877a_0cb95a840207.slice/crio-a1ea034ee151661ba62ee5ccf83de4d20ef0dac6275f5d17861b06533aacb64b WatchSource:0}: Error finding container a1ea034ee151661ba62ee5ccf83de4d20ef0dac6275f5d17861b06533aacb64b: Status 404 returned error can't find the container with id a1ea034ee151661ba62ee5ccf83de4d20ef0dac6275f5d17861b06533aacb64b Nov 02 17:44:02.429960 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.429881 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6"] Nov 02 17:44:02.434712 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.434691 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.447314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.447289 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8p5bl\"" Nov 02 17:44:02.450034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.450003 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6"] Nov 02 17:44:02.495311 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.495278 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a109e0b-196a-4463-b765-169521e65070-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-gt9f6\" (UID: \"9a109e0b-196a-4463-b765-169521e65070\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.495487 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.495423 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbmw\" (UniqueName: \"kubernetes.io/projected/9a109e0b-196a-4463-b765-169521e65070-kube-api-access-rfbmw\") pod \"cert-manager-cainjector-5c57bbd4fc-gt9f6\" (UID: \"9a109e0b-196a-4463-b765-169521e65070\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.596746 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.596712 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a109e0b-196a-4463-b765-169521e65070-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-gt9f6\" (UID: \"9a109e0b-196a-4463-b765-169521e65070\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.596917 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.596863 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbmw\" (UniqueName: \"kubernetes.io/projected/9a109e0b-196a-4463-b765-169521e65070-kube-api-access-rfbmw\") pod \"cert-manager-cainjector-5c57bbd4fc-gt9f6\" (UID: \"9a109e0b-196a-4463-b765-169521e65070\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.609476 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.609412 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a109e0b-196a-4463-b765-169521e65070-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-gt9f6\" (UID: \"9a109e0b-196a-4463-b765-169521e65070\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.609476 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.609448 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbmw\" (UniqueName: \"kubernetes.io/projected/9a109e0b-196a-4463-b765-169521e65070-kube-api-access-rfbmw\") pod \"cert-manager-cainjector-5c57bbd4fc-gt9f6\" (UID: \"9a109e0b-196a-4463-b765-169521e65070\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.746353 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.746268 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" Nov 02 17:44:02.854246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.854123 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" event={"ID":"52bf20cf-0da5-496a-ab03-ff53cfc7587f","Type":"ContainerStarted","Data":"7756df67303882fd44bc94e755002c1f69a73787135f96d51d040efc0bc360ed"} Nov 02 17:44:02.855954 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:02.855925 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" event={"ID":"80de8a96-1a74-4e82-877a-0cb95a840207","Type":"ContainerStarted","Data":"a1ea034ee151661ba62ee5ccf83de4d20ef0dac6275f5d17861b06533aacb64b"} Nov 02 17:44:03.058979 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:03.058945 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6"] Nov 02 17:44:03.062616 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:03.062583 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a109e0b_196a_4463_b765_169521e65070.slice/crio-fb671a073d5ced1999cd8784993d3f5e431b8f1f441174f12ef6517531125ef3 WatchSource:0}: Error finding container fb671a073d5ced1999cd8784993d3f5e431b8f1f441174f12ef6517531125ef3: Status 404 returned error can't find the container with id fb671a073d5ced1999cd8784993d3f5e431b8f1f441174f12ef6517531125ef3 Nov 02 17:44:03.864138 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:03.864063 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" event={"ID":"9a109e0b-196a-4463-b765-169521e65070","Type":"ContainerStarted","Data":"fb671a073d5ced1999cd8784993d3f5e431b8f1f441174f12ef6517531125ef3"} Nov 02 17:44:05.876501 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:05.876461 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" event={"ID":"52bf20cf-0da5-496a-ab03-ff53cfc7587f","Type":"ContainerStarted","Data":"c7c88adc729b6f0a8b0534c1f36691d786dd53b6a22d09b985afcee39810bff9"} Nov 02 17:44:05.878268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:05.878241 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" event={"ID":"80de8a96-1a74-4e82-877a-0cb95a840207","Type":"ContainerStarted","Data":"e2bb714cdb424b6bb57582c82400eb449cd645c2d0c2e52f1240aeb44e9a760a"} Nov 02 17:44:05.878410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:05.878328 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:06.019461 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:06.019409 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" podStartSLOduration=1.482714693 podStartE2EDuration="5.019391522s" totalImagesPullingTime="3.536676829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:06.018876999 +0000 UTC m=+206.117905191" watchObservedRunningTime="2026-11-02 17:44:06.019391522 +0000 UTC m=+206.118419719" Nov 02 17:44:06.887506 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:06.887472 6343 generic.go:356] "Generic (PLEG): container finished" podID="52bf20cf-0da5-496a-ab03-ff53cfc7587f" containerID="c7c88adc729b6f0a8b0534c1f36691d786dd53b6a22d09b985afcee39810bff9" exitCode=0 Nov 02 17:44:06.887982 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:06.887550 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" event={"ID":"52bf20cf-0da5-496a-ab03-ff53cfc7587f","Type":"ContainerDied","Data":"c7c88adc729b6f0a8b0534c1f36691d786dd53b6a22d09b985afcee39810bff9"} Nov 02 17:44:06.894743 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:06.894690 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" event={"ID":"9a109e0b-196a-4463-b765-169521e65070","Type":"ContainerStarted","Data":"0d731171bd40b27b24c3a09c81f2946340b7cbc457d8caa6bd55b42a70180c2f"} Nov 02 17:44:07.109283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:07.109138 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gt9f6" podStartSLOduration=1.8678613 podStartE2EDuration="5.109121834s" totalImagesPullingTime="3.241260534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:07.108991532 +0000 UTC m=+207.208019728" watchObservedRunningTime="2026-11-02 17:44:07.109121834 +0000 UTC m=+207.208150031" Nov 02 17:44:07.900525 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:07.900491 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" event={"ID":"52bf20cf-0da5-496a-ab03-ff53cfc7587f","Type":"ContainerStarted","Data":"b3d9d642e9ef79bc7775f395e976c078969540b2b55dc8e0bff3de9236a640af"} Nov 02 17:44:07.981802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:07.981747 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" podStartSLOduration=2.499786028 podStartE2EDuration="6.981734603s" totalImagesPullingTime="3.8693623s" totalInitContainerRuntime="612.586275ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:07.964862445 +0000 UTC m=+208.063890653" watchObservedRunningTime="2026-11-02 17:44:07.981734603 +0000 UTC m=+208.080762791" Nov 02 17:44:08.904801 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:08.904774 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:10.657431 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.657397 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-sjt6d"] Nov 02 17:44:10.660931 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.660909 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.672583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.672561 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dxhrz\"" Nov 02 17:44:10.695110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.695080 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-sjt6d"] Nov 02 17:44:10.772644 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.772606 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rqf\" (UniqueName: \"kubernetes.io/projected/0abf131f-7ea0-4ba2-8dd5-728798b362a9-kube-api-access-98rqf\") pod \"cert-manager-597b95cb84-sjt6d\" (UID: \"0abf131f-7ea0-4ba2-8dd5-728798b362a9\") " pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.772821 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.772660 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0abf131f-7ea0-4ba2-8dd5-728798b362a9-bound-sa-token\") pod \"cert-manager-597b95cb84-sjt6d\" (UID: \"0abf131f-7ea0-4ba2-8dd5-728798b362a9\") " pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.873914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.873877 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rqf\" (UniqueName: \"kubernetes.io/projected/0abf131f-7ea0-4ba2-8dd5-728798b362a9-kube-api-access-98rqf\") pod \"cert-manager-597b95cb84-sjt6d\" (UID: \"0abf131f-7ea0-4ba2-8dd5-728798b362a9\") " pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.874147 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.873925 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0abf131f-7ea0-4ba2-8dd5-728798b362a9-bound-sa-token\") pod \"cert-manager-597b95cb84-sjt6d\" (UID: \"0abf131f-7ea0-4ba2-8dd5-728798b362a9\") " pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.888906 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.888882 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rqf\" (UniqueName: \"kubernetes.io/projected/0abf131f-7ea0-4ba2-8dd5-728798b362a9-kube-api-access-98rqf\") pod \"cert-manager-597b95cb84-sjt6d\" (UID: \"0abf131f-7ea0-4ba2-8dd5-728798b362a9\") " pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.890141 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.890118 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0abf131f-7ea0-4ba2-8dd5-728798b362a9-bound-sa-token\") pod \"cert-manager-597b95cb84-sjt6d\" (UID: \"0abf131f-7ea0-4ba2-8dd5-728798b362a9\") " pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:10.972959 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:10.972887 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-sjt6d" Nov 02 17:44:11.323440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:11.323372 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-sjt6d"] Nov 02 17:44:11.331019 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:11.330994 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0abf131f_7ea0_4ba2_8dd5_728798b362a9.slice/crio-335751723d786eb88df7ad871ee718829411fd39751dc7635ff40edc9f10febe WatchSource:0}: Error finding container 335751723d786eb88df7ad871ee718829411fd39751dc7635ff40edc9f10febe: Status 404 returned error can't find the container with id 335751723d786eb88df7ad871ee718829411fd39751dc7635ff40edc9f10febe Nov 02 17:44:11.896940 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:11.896913 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" Nov 02 17:44:11.918440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:11.918410 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-sjt6d" event={"ID":"0abf131f-7ea0-4ba2-8dd5-728798b362a9","Type":"ContainerStarted","Data":"5d0855c91b33f59ca03370e29164a978198486341b4edfb122f4ffa438755325"} Nov 02 17:44:11.918598 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:11.918445 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-sjt6d" event={"ID":"0abf131f-7ea0-4ba2-8dd5-728798b362a9","Type":"ContainerStarted","Data":"335751723d786eb88df7ad871ee718829411fd39751dc7635ff40edc9f10febe"} Nov 02 17:44:11.975833 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:11.975787 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-sjt6d" podStartSLOduration=1.975773477 podStartE2EDuration="1.975773477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:11.974772635 +0000 UTC m=+212.073800829" watchObservedRunningTime="2026-11-02 17:44:11.975773477 +0000 UTC m=+212.074801669" Nov 02 17:44:19.911313 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:19.911284 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88" Nov 02 17:44:24.197400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.197364 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2"] Nov 02 17:44:24.199944 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.199924 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.216649 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.216629 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 17:44:24.217845 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.217832 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j8t8g\"" Nov 02 17:44:24.225016 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.224989 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2"] Nov 02 17:44:24.294838 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.294813 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:44:24.294838 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.294814 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 17:44:24.295088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.295073 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:44:24.300547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.300524 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.300620 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.300591 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.300681 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.300666 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfgs\" (UniqueName: \"kubernetes.io/projected/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-kube-api-access-tlfgs\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.401607 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.401565 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfgs\" (UniqueName: \"kubernetes.io/projected/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-kube-api-access-tlfgs\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.401816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.401628 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.401816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.401670 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.402305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.402282 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.403987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.403965 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.428241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.428210 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfgs\" (UniqueName: \"kubernetes.io/projected/895bdbc5-eb39-4ede-9b4b-39d32b1445b6-kube-api-access-tlfgs\") pod \"kuadrant-console-plugin-66c4fb575d-vqfj2\" (UID: \"895bdbc5-eb39-4ede-9b4b-39d32b1445b6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.510027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.509942 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" Nov 02 17:44:24.799941 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.799862 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2"] Nov 02 17:44:24.803070 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:24.803025 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod895bdbc5_eb39_4ede_9b4b_39d32b1445b6.slice/crio-63ae75fb7736d8971c49f1b26c86b42592c3b88383cd2c0fa8fa4d4f54de8f57 WatchSource:0}: Error finding container 63ae75fb7736d8971c49f1b26c86b42592c3b88383cd2c0fa8fa4d4f54de8f57: Status 404 returned error can't find the container with id 63ae75fb7736d8971c49f1b26c86b42592c3b88383cd2c0fa8fa4d4f54de8f57 Nov 02 17:44:24.972571 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:24.972531 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" event={"ID":"895bdbc5-eb39-4ede-9b4b-39d32b1445b6","Type":"ContainerStarted","Data":"63ae75fb7736d8971c49f1b26c86b42592c3b88383cd2c0fa8fa4d4f54de8f57"} Nov 02 17:44:29.265081 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:44:29.265025 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:44:30.002763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:30.002727 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" event={"ID":"895bdbc5-eb39-4ede-9b4b-39d32b1445b6","Type":"ContainerStarted","Data":"e12d149e8e4143e03364b63798cd072addb46a60cec49b39e36dd7b12012b37b"} Nov 02 17:44:30.022737 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:30.022683 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vqfj2" podStartSLOduration=1.2606366279999999 podStartE2EDuration="6.022665117s" totalImagesPullingTime="4.762028489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:30.022124538 +0000 UTC m=+230.121152733" watchObservedRunningTime="2026-11-02 17:44:30.022665117 +0000 UTC m=+230.121693308" Nov 02 17:44:35.582364 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.582328 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7"] Nov 02 17:44:35.587546 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.587528 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.609721 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.609688 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7"] Nov 02 17:44:35.621430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.621411 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:44:35.621618 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.621600 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:44:35.649511 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.649487 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:44:35.649619 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.649516 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c45v5\"" Nov 02 17:44:35.649671 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.649658 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 17:44:35.681060 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.681016 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1108ab25-d875-4900-bd00-809b09003811-tls-certs\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.681178 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.681068 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbxq\" (UniqueName: \"kubernetes.io/projected/1108ab25-d875-4900-bd00-809b09003811-kube-api-access-rmbxq\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.681341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.681145 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1108ab25-d875-4900-bd00-809b09003811-tmp\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.782465 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.782431 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1108ab25-d875-4900-bd00-809b09003811-tls-certs\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.782627 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.782484 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbxq\" (UniqueName: \"kubernetes.io/projected/1108ab25-d875-4900-bd00-809b09003811-kube-api-access-rmbxq\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.782627 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.782512 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1108ab25-d875-4900-bd00-809b09003811-tmp\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.785569 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.785538 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1108ab25-d875-4900-bd00-809b09003811-tls-certs\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.786974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.786952 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1108ab25-d875-4900-bd00-809b09003811-tmp\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.802022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.801995 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbxq\" (UniqueName: \"kubernetes.io/projected/1108ab25-d875-4900-bd00-809b09003811-kube-api-access-rmbxq\") pod \"kube-auth-proxy-555d4cc554-wmzt7\" (UID: \"1108ab25-d875-4900-bd00-809b09003811\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:35.900648 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:35.900558 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" Nov 02 17:44:36.194189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:36.194143 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7"] Nov 02 17:44:36.196988 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:36.196962 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1108ab25_d875_4900_bd00_809b09003811.slice/crio-fba7a00101d697f1e620acecf21ce6e7e1fcb8ffd556e9ca4d02d1d391c89cbb WatchSource:0}: Error finding container fba7a00101d697f1e620acecf21ce6e7e1fcb8ffd556e9ca4d02d1d391c89cbb: Status 404 returned error can't find the container with id fba7a00101d697f1e620acecf21ce6e7e1fcb8ffd556e9ca4d02d1d391c89cbb Nov 02 17:44:37.036084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:37.035983 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" event={"ID":"1108ab25-d875-4900-bd00-809b09003811","Type":"ContainerStarted","Data":"fba7a00101d697f1e620acecf21ce6e7e1fcb8ffd556e9ca4d02d1d391c89cbb"} Nov 02 17:44:38.142076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.140752 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zd2fn"] Nov 02 17:44:38.236594 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.235798 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zd2fn"] Nov 02 17:44:38.236594 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.235945 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:44:38.254660 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.254622 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lhbkr\"" Nov 02 17:44:38.304777 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.304692 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmffd\" (UniqueName: \"kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd\") pod \"authorino-57f7f8659f-zd2fn\" (UID: \"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae\") " pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:44:38.405926 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.405803 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmffd\" (UniqueName: \"kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd\") pod \"authorino-57f7f8659f-zd2fn\" (UID: \"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae\") " pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:44:38.419302 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.419276 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmffd\" (UniqueName: \"kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd\") pod \"authorino-57f7f8659f-zd2fn\" (UID: \"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae\") " pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:44:38.572212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.572182 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:44:38.586219 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.586192 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6"] Nov 02 17:44:38.605830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.605808 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.609270 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.609248 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6"] Nov 02 17:44:38.616690 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.616651 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9m5h8\"" Nov 02 17:44:38.616787 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.616753 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 17:44:38.616862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.616847 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 17:44:38.621706 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.621681 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 17:44:38.621795 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.621753 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 17:44:38.621914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.621897 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:44:38.708935 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.708904 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773f1045-d567-4a52-b84d-cd01dcf02b94-metrics-certs\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.709087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.708950 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/773f1045-d567-4a52-b84d-cd01dcf02b94-manager-config\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.709132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.709091 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/773f1045-d567-4a52-b84d-cd01dcf02b94-cert\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.709166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.709144 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcnj\" (UniqueName: \"kubernetes.io/projected/773f1045-d567-4a52-b84d-cd01dcf02b94-kube-api-access-bwcnj\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.810326 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.810298 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/773f1045-d567-4a52-b84d-cd01dcf02b94-manager-config\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.810454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.810332 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/773f1045-d567-4a52-b84d-cd01dcf02b94-cert\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.810454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.810366 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcnj\" (UniqueName: \"kubernetes.io/projected/773f1045-d567-4a52-b84d-cd01dcf02b94-kube-api-access-bwcnj\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.810454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.810412 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773f1045-d567-4a52-b84d-cd01dcf02b94-metrics-certs\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.811009 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.810982 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/773f1045-d567-4a52-b84d-cd01dcf02b94-manager-config\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.812836 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.812816 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773f1045-d567-4a52-b84d-cd01dcf02b94-metrics-certs\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.812943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.812822 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/773f1045-d567-4a52-b84d-cd01dcf02b94-cert\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.828255 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.828235 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcnj\" (UniqueName: \"kubernetes.io/projected/773f1045-d567-4a52-b84d-cd01dcf02b94-kube-api-access-bwcnj\") pod \"jobset-controller-manager-7b476c9985-ttxw6\" (UID: \"773f1045-d567-4a52-b84d-cd01dcf02b94\") " pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:38.884071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.884031 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zd2fn"] Nov 02 17:44:38.888099 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:38.888075 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa7adc0_ae8d_49af_93df_bb6d0bd54eae.slice/crio-9a15aaad9506e9c7fbd60307f118eca253c4cf5d2af827d1dc709f2e8f0e7682 WatchSource:0}: Error finding container 9a15aaad9506e9c7fbd60307f118eca253c4cf5d2af827d1dc709f2e8f0e7682: Status 404 returned error can't find the container with id 9a15aaad9506e9c7fbd60307f118eca253c4cf5d2af827d1dc709f2e8f0e7682 Nov 02 17:44:38.977226 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:38.977180 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:39.046309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:39.046285 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" event={"ID":"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae","Type":"ContainerStarted","Data":"9a15aaad9506e9c7fbd60307f118eca253c4cf5d2af827d1dc709f2e8f0e7682"} Nov 02 17:44:39.396003 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:39.395921 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6"] Nov 02 17:44:39.398517 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:44:39.398481 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773f1045_d567_4a52_b84d_cd01dcf02b94.slice/crio-9515be9621dfd165cb107b25406048acdbf7a65a36ed9e89d0c27d698624cae8 WatchSource:0}: Error finding container 9515be9621dfd165cb107b25406048acdbf7a65a36ed9e89d0c27d698624cae8: Status 404 returned error can't find the container with id 9515be9621dfd165cb107b25406048acdbf7a65a36ed9e89d0c27d698624cae8 Nov 02 17:44:40.053022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:40.052572 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" event={"ID":"1108ab25-d875-4900-bd00-809b09003811","Type":"ContainerStarted","Data":"d10c88ca9b7b36f2b7bf4e04319b4f235666c66b187b50e033e305c6f1d43310"} Nov 02 17:44:40.054357 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:40.054330 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" event={"ID":"773f1045-d567-4a52-b84d-cd01dcf02b94","Type":"ContainerStarted","Data":"9515be9621dfd165cb107b25406048acdbf7a65a36ed9e89d0c27d698624cae8"} Nov 02 17:44:43.070642 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:43.070603 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" event={"ID":"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae","Type":"ContainerStarted","Data":"a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174"} Nov 02 17:44:43.072135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:43.072099 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" event={"ID":"773f1045-d567-4a52-b84d-cd01dcf02b94","Type":"ContainerStarted","Data":"d75dd01f4fb016af182e6ae3f5ab6ea666c017c8de6dac9c8328b1aa70686fd5"} Nov 02 17:44:43.072475 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:43.072459 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:43.087996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:43.087954 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-555d4cc554-wmzt7" podStartSLOduration=5.170327657 podStartE2EDuration="8.087942658s" totalImagesPullingTime="2.917615001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:40.104238113 +0000 UTC m=+240.203266309" watchObservedRunningTime="2026-11-02 17:44:43.087942658 +0000 UTC m=+243.186970889" Nov 02 17:44:43.088130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:43.088031 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" podStartSLOduration=1.860994912 podStartE2EDuration="5.08802776s" totalImagesPullingTime="3.227032848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:43.086982387 +0000 UTC m=+243.186010584" watchObservedRunningTime="2026-11-02 17:44:43.08802776 +0000 UTC m=+243.187055955" Nov 02 17:44:43.108811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:43.108758 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" podStartSLOduration=2.04830392 podStartE2EDuration="5.108741307s" totalImagesPullingTime="3.060437387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:43.107224014 +0000 UTC m=+243.206252201" watchObservedRunningTime="2026-11-02 17:44:43.108741307 +0000 UTC m=+243.207769503" Nov 02 17:44:55.085187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:44:55.085155 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7b476c9985-ttxw6" Nov 02 17:44:59.260274 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:44:59.260228 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:45:10.348785 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:10.348752 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zd2fn"] Nov 02 17:45:10.349280 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:10.348940 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" podUID="8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" containerName="authorino" containerID="cri-o://a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174" gracePeriod=30 Nov 02 17:45:10.594112 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:10.594089 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:45:10.698611 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:10.698553 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd\" (UniqueName: \"kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd\") pod \"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae\" (UID: \"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae\") " Nov 02 17:45:10.700655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:10.700633 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd" pod "8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" (UID: "8fa7adc0-ae8d-49af-93df-bb6d0bd54eae"). InnerVolumeSpecName "kube-api-access-vmffd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:10.799242 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:10.799218 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmffd\" (UniqueName: \"kubernetes.io/projected/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae-kube-api-access-vmffd\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:11.191965 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.191930 6343 generic.go:356] "Generic (PLEG): container finished" podID="8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" containerID="a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174" exitCode=0 Nov 02 17:45:11.192148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.191974 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" Nov 02 17:45:11.192148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.192022 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" event={"ID":"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae","Type":"ContainerDied","Data":"a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174"} Nov 02 17:45:11.192148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.192081 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zd2fn" event={"ID":"8fa7adc0-ae8d-49af-93df-bb6d0bd54eae","Type":"ContainerDied","Data":"9a15aaad9506e9c7fbd60307f118eca253c4cf5d2af827d1dc709f2e8f0e7682"} Nov 02 17:45:11.192148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.192100 6343 scope.go:122] "RemoveContainer" containerID="a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174" Nov 02 17:45:11.200856 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.200836 6343 scope.go:122] "RemoveContainer" containerID="a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174" Nov 02 17:45:11.201155 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:45:11.201138 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174\": container with ID starting with a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174 not found: ID does not exist" containerID="a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174" Nov 02 17:45:11.201208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.201164 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174"} err="failed to get container status \"a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174\": rpc error: code = NotFound desc = could not find container \"a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174\": container with ID starting with a5cb46e555d7794a71ae08592cb392bcf205848aa15a29050b495184524d0174 not found: ID does not exist" Nov 02 17:45:11.212912 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.212891 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zd2fn"] Nov 02 17:45:11.218703 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:11.218681 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zd2fn"] Nov 02 17:45:12.018293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:12.018261 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" path="/var/lib/kubelet/pods/8fa7adc0-ae8d-49af-93df-bb6d0bd54eae/volumes" Nov 02 17:45:29.265591 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:45:29.265466 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:45:32.422002 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.421965 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn"] Nov 02 17:45:32.422450 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.422431 6343 cpu_manager.go:414] "containerMap: removing container" podUID="8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" containerName="authorino" Nov 02 17:45:32.422450 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.422448 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" containerName="authorino" Nov 02 17:45:32.422534 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.422523 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa7adc0-ae8d-49af-93df-bb6d0bd54eae" containerName="authorino" Nov 02 17:45:32.425542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.425524 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.434941 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.434897 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:45:32.436416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.436395 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:45:32.437597 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.437579 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn"] Nov 02 17:45:32.458031 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.456629 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 17:45:32.458031 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.456848 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2tm4x\"" Nov 02 17:45:32.581502 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.581462 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sdrxn\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.581502 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.581510 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq69h\" (UniqueName: \"kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h\") pod \"llmisvc-controller-manager-869d5dcc5d-sdrxn\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.682495 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.682395 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sdrxn\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.682495 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.682461 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq69h\" (UniqueName: \"kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h\") pod \"llmisvc-controller-manager-869d5dcc5d-sdrxn\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.684843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.684815 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sdrxn\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.695806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.695777 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq69h\" (UniqueName: \"kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h\") pod \"llmisvc-controller-manager-869d5dcc5d-sdrxn\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:32.739371 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:32.739331 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:33.014735 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:33.014560 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn"] Nov 02 17:45:33.017784 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:45:33.017757 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48e119c1_7b79_4ab4_a495_8aef164495d7.slice/crio-c1951cad852ad28798b205d3a0cc9fe36df8c3bdcfb67b595cd21d30d47828ef WatchSource:0}: Error finding container c1951cad852ad28798b205d3a0cc9fe36df8c3bdcfb67b595cd21d30d47828ef: Status 404 returned error can't find the container with id c1951cad852ad28798b205d3a0cc9fe36df8c3bdcfb67b595cd21d30d47828ef Nov 02 17:45:33.304976 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:33.304884 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" event={"ID":"48e119c1-7b79-4ab4-a495-8aef164495d7","Type":"ContainerStarted","Data":"c1951cad852ad28798b205d3a0cc9fe36df8c3bdcfb67b595cd21d30d47828ef"} Nov 02 17:45:36.319362 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:36.319325 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" event={"ID":"48e119c1-7b79-4ab4-a495-8aef164495d7","Type":"ContainerStarted","Data":"b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd"} Nov 02 17:45:36.319761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:36.319401 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:45:36.340779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:36.340719 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" podStartSLOduration=1.928330458 podStartE2EDuration="4.340705732s" totalImagesPullingTime="2.412375274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:36.339684098 +0000 UTC m=+296.438712294" watchObservedRunningTime="2026-11-02 17:45:36.340705732 +0000 UTC m=+296.439733926" Nov 02 17:45:40.059488 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:40.059458 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:45:40.059951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:45:40.059731 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:45:59.245102 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:45:59.245063 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:46:07.326083 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:07.326029 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:46:29.245348 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:46:29.245313 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:46:30.410876 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.410842 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-klbfw"] Nov 02 17:46:30.414123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.414106 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.419999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.419980 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lhbkr\"" Nov 02 17:46:30.424202 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.424180 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-klbfw"] Nov 02 17:46:30.452625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.452595 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-klbfw"] Nov 02 17:46:30.452805 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:46:30.452787 6343 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-27x2n tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-27x2n tls-cert]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-klbfw" podUID="7910ea32-54c5-41be-a008-fc9b22b963db" Nov 02 17:46:30.488148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.488122 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert\") pod \"authorino-6dd76875b5-klbfw\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.488243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.488151 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27x2n\" (UniqueName: \"kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n\") pod \"authorino-6dd76875b5-klbfw\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.519974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.519953 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 17:46:30.561106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.561083 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.567738 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.567720 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.588741 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.588712 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert\") pod \"authorino-6dd76875b5-klbfw\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.588741 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.588744 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27x2n\" (UniqueName: \"kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n\") pod \"authorino-6dd76875b5-klbfw\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.591226 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.591201 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert\") pod \"authorino-6dd76875b5-klbfw\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.596928 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.596905 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27x2n\" (UniqueName: \"kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n\") pod \"authorino-6dd76875b5-klbfw\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:30.619015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.618977 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66989884-r7kj9"] Nov 02 17:46:30.622522 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.622498 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.626787 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.626758 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:46:30.626787 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.626772 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 17:46:30.627756 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.627741 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-szhm5\"" Nov 02 17:46:30.627843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.627803 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:46:30.627843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.627806 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:46:30.635416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.635389 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66989884-r7kj9"] Nov 02 17:46:30.689370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.689289 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert\" (UniqueName: \"kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert\") pod \"7910ea32-54c5-41be-a008-fc9b22b963db\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " Nov 02 17:46:30.689370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.689328 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n\" (UniqueName: \"kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n\") pod \"7910ea32-54c5-41be-a008-fc9b22b963db\" (UID: \"7910ea32-54c5-41be-a008-fc9b22b963db\") " Nov 02 17:46:30.689573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.689400 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.689573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.689426 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.689573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.689445 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dl9\" (UniqueName: \"kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.689741 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.689716 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.691358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.691331 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert" pod "7910ea32-54c5-41be-a008-fc9b22b963db" (UID: "7910ea32-54c5-41be-a008-fc9b22b963db"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:30.691549 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.691528 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n" pod "7910ea32-54c5-41be-a008-fc9b22b963db" (UID: "7910ea32-54c5-41be-a008-fc9b22b963db"). InnerVolumeSpecName "kube-api-access-27x2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:30.790875 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.790838 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.791087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.790886 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.791087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.790912 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.791087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.791031 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dl9\" (UniqueName: \"kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.791252 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.791148 6343 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7910ea32-54c5-41be-a008-fc9b22b963db-tls-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:46:30.791252 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.791161 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27x2n\" (UniqueName: \"kubernetes.io/projected/7910ea32-54c5-41be-a008-fc9b22b963db-kube-api-access-27x2n\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:46:30.791546 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.791521 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.793412 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.793392 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.793487 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.793441 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.803326 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.803302 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dl9\" (UniqueName: \"kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9\") pod \"maas-api-66989884-r7kj9\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:30.942460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:30.942375 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:31.215586 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:31.215508 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66989884-r7kj9"] Nov 02 17:46:31.218454 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:46:31.218418 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1d6f08_e3ad_49c8_8eb8_bf818b54a7c8.slice/crio-a01d4dbb00b4dbab6303eb83baa65efefef09c7dcb1f10ce5c4ff2d736e65876 WatchSource:0}: Error finding container a01d4dbb00b4dbab6303eb83baa65efefef09c7dcb1f10ce5c4ff2d736e65876: Status 404 returned error can't find the container with id a01d4dbb00b4dbab6303eb83baa65efefef09c7dcb1f10ce5c4ff2d736e65876 Nov 02 17:46:31.219655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:31.219637 6343 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:46:31.566190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:31.566100 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" event={"ID":"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8","Type":"ContainerStarted","Data":"a01d4dbb00b4dbab6303eb83baa65efefef09c7dcb1f10ce5c4ff2d736e65876"} Nov 02 17:46:31.566190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:31.566155 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-klbfw" Nov 02 17:46:31.600458 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:31.600413 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-klbfw"] Nov 02 17:46:31.601921 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:31.601894 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-klbfw"] Nov 02 17:46:32.021433 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:32.021398 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7910ea32-54c5-41be-a008-fc9b22b963db" path="/var/lib/kubelet/pods/7910ea32-54c5-41be-a008-fc9b22b963db/volumes" Nov 02 17:46:33.690347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.690307 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ckhld"] Nov 02 17:46:33.694568 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.694540 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:33.706261 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.706233 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ckhld"] Nov 02 17:46:33.715836 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.715811 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmvl\" (UniqueName: \"kubernetes.io/projected/6c87c46e-5bc2-45d9-a00d-a880a93fa01d-kube-api-access-6bmvl\") pod \"llmisvc-controller-manager-6466c554d8-ckhld\" (UID: \"6c87c46e-5bc2-45d9-a00d-a880a93fa01d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:33.715986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.715843 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87c46e-5bc2-45d9-a00d-a880a93fa01d-cert\") pod \"llmisvc-controller-manager-6466c554d8-ckhld\" (UID: \"6c87c46e-5bc2-45d9-a00d-a880a93fa01d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:33.817282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.817248 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmvl\" (UniqueName: \"kubernetes.io/projected/6c87c46e-5bc2-45d9-a00d-a880a93fa01d-kube-api-access-6bmvl\") pod \"llmisvc-controller-manager-6466c554d8-ckhld\" (UID: \"6c87c46e-5bc2-45d9-a00d-a880a93fa01d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:33.817282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.817287 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87c46e-5bc2-45d9-a00d-a880a93fa01d-cert\") pod \"llmisvc-controller-manager-6466c554d8-ckhld\" (UID: \"6c87c46e-5bc2-45d9-a00d-a880a93fa01d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:33.820177 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.820151 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87c46e-5bc2-45d9-a00d-a880a93fa01d-cert\") pod \"llmisvc-controller-manager-6466c554d8-ckhld\" (UID: \"6c87c46e-5bc2-45d9-a00d-a880a93fa01d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:33.827076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:33.827037 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmvl\" (UniqueName: \"kubernetes.io/projected/6c87c46e-5bc2-45d9-a00d-a880a93fa01d-kube-api-access-6bmvl\") pod \"llmisvc-controller-manager-6466c554d8-ckhld\" (UID: \"6c87c46e-5bc2-45d9-a00d-a880a93fa01d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:34.008369 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:34.008278 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:34.283170 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:34.278989 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ckhld"] Nov 02 17:46:34.286074 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:46:34.286032 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c87c46e_5bc2_45d9_a00d_a880a93fa01d.slice/crio-5eac62f634ae3bc87fd4483b8e365b4541a5c18960addfd5f83373df00da4e81 WatchSource:0}: Error finding container 5eac62f634ae3bc87fd4483b8e365b4541a5c18960addfd5f83373df00da4e81: Status 404 returned error can't find the container with id 5eac62f634ae3bc87fd4483b8e365b4541a5c18960addfd5f83373df00da4e81 Nov 02 17:46:34.581439 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:34.581406 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" event={"ID":"6c87c46e-5bc2-45d9-a00d-a880a93fa01d","Type":"ContainerStarted","Data":"5eac62f634ae3bc87fd4483b8e365b4541a5c18960addfd5f83373df00da4e81"} Nov 02 17:46:34.582947 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:34.582922 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" event={"ID":"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8","Type":"ContainerStarted","Data":"ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8"} Nov 02 17:46:34.583086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:34.583064 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:34.602251 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:34.602209 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" podStartSLOduration=2.241516103 podStartE2EDuration="4.602196992s" totalImagesPullingTime="2.360680889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:34.601100592 +0000 UTC m=+354.700128785" watchObservedRunningTime="2026-11-02 17:46:34.602196992 +0000 UTC m=+354.701225183" Nov 02 17:46:35.588559 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:35.588523 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" event={"ID":"6c87c46e-5bc2-45d9-a00d-a880a93fa01d","Type":"ContainerStarted","Data":"3dc90b367d365289f7adfcbd0659fd64f51ad8949d6cbb15f2a25e5437da4e73"} Nov 02 17:46:35.605162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:35.605112 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" podStartSLOduration=2.20477267 podStartE2EDuration="2.605096351s" totalImagesPullingTime="400.323681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:35.604700183 +0000 UTC m=+355.703728377" watchObservedRunningTime="2026-11-02 17:46:35.605096351 +0000 UTC m=+355.704124547" Nov 02 17:46:36.593474 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:36.593445 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:46:40.594412 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:46:40.594385 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 17:46:59.245755 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:46:59.245721 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:47:07.599114 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:07.599085 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ckhld" Nov 02 17:47:07.644725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:07.644691 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn"] Nov 02 17:47:07.644971 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:07.644930 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" podUID="48e119c1-7b79-4ab4-a495-8aef164495d7" containerName="manager" containerID="cri-o://b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd" gracePeriod=30 Nov 02 17:47:07.887692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:07.887669 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:47:08.015307 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.015280 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h\" (UniqueName: \"kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h\") pod \"48e119c1-7b79-4ab4-a495-8aef164495d7\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " Nov 02 17:47:08.015497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.015381 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert\" (UniqueName: \"kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert\") pod \"48e119c1-7b79-4ab4-a495-8aef164495d7\" (UID: \"48e119c1-7b79-4ab4-a495-8aef164495d7\") " Nov 02 17:47:08.017706 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.017678 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert" pod "48e119c1-7b79-4ab4-a495-8aef164495d7" (UID: "48e119c1-7b79-4ab4-a495-8aef164495d7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:08.017815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.017799 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h" pod "48e119c1-7b79-4ab4-a495-8aef164495d7" (UID: "48e119c1-7b79-4ab4-a495-8aef164495d7"). InnerVolumeSpecName "kube-api-access-gq69h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:08.116453 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.116420 6343 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e119c1-7b79-4ab4-a495-8aef164495d7-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:08.116453 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.116448 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq69h\" (UniqueName: \"kubernetes.io/projected/48e119c1-7b79-4ab4-a495-8aef164495d7-kube-api-access-gq69h\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:08.741854 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.741816 6343 generic.go:356] "Generic (PLEG): container finished" podID="48e119c1-7b79-4ab4-a495-8aef164495d7" containerID="b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd" exitCode=0 Nov 02 17:47:08.742347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.741864 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" event={"ID":"48e119c1-7b79-4ab4-a495-8aef164495d7","Type":"ContainerDied","Data":"b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd"} Nov 02 17:47:08.742347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.741880 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" Nov 02 17:47:08.742347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.741947 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn" event={"ID":"48e119c1-7b79-4ab4-a495-8aef164495d7","Type":"ContainerDied","Data":"c1951cad852ad28798b205d3a0cc9fe36df8c3bdcfb67b595cd21d30d47828ef"} Nov 02 17:47:08.742347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.741978 6343 scope.go:122] "RemoveContainer" containerID="b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd" Nov 02 17:47:08.750472 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.750451 6343 scope.go:122] "RemoveContainer" containerID="b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd" Nov 02 17:47:08.750724 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:47:08.750708 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd\": container with ID starting with b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd not found: ID does not exist" containerID="b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd" Nov 02 17:47:08.750764 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.750731 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd"} err="failed to get container status \"b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd\": rpc error: code = NotFound desc = could not find container \"b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd\": container with ID starting with b005cf38c2d8bb92df61f63e15cf0139ae629f04390cce5f72ccb00a84c2d2cd not found: ID does not exist" Nov 02 17:47:08.760355 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.760332 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn"] Nov 02 17:47:08.763809 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:08.763790 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn"] Nov 02 17:47:10.018358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:47:10.018324 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e119c1-7b79-4ab4-a495-8aef164495d7" path="/var/lib/kubelet/pods/48e119c1-7b79-4ab4-a495-8aef164495d7/volumes" Nov 02 17:47:29.246283 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:47:29.246242 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:47:59.245090 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:47:59.245031 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:48:01.349033 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.349003 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c"] Nov 02 17:48:01.349672 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.349595 6343 cpu_manager.go:414] "containerMap: removing container" podUID="48e119c1-7b79-4ab4-a495-8aef164495d7" containerName="manager" Nov 02 17:48:01.349672 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.349615 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e119c1-7b79-4ab4-a495-8aef164495d7" containerName="manager" Nov 02 17:48:01.349804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.349720 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e119c1-7b79-4ab4-a495-8aef164495d7" containerName="manager" Nov 02 17:48:01.353210 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.353187 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.368468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.368436 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c"] Nov 02 17:48:01.434539 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.434508 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:48:01.450784 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.450759 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zplmb\"" Nov 02 17:48:01.479258 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479220 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479431 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479274 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479475 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479448 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479514 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479486 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479514 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479509 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttw5\" (UniqueName: \"kubernetes.io/projected/72508624-36f4-4d4a-992b-37271137a520-kube-api-access-sttw5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479661 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479635 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72508624-36f4-4d4a-992b-37271137a520-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479712 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479697 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479760 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479721 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72508624-36f4-4d4a-992b-37271137a520-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.479760 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.479740 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72508624-36f4-4d4a-992b-37271137a520-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580297 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580254 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580297 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580300 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580322 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttw5\" (UniqueName: \"kubernetes.io/projected/72508624-36f4-4d4a-992b-37271137a520-kube-api-access-sttw5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580377 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72508624-36f4-4d4a-992b-37271137a520-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580407 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580429 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72508624-36f4-4d4a-992b-37271137a520-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580454 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72508624-36f4-4d4a-992b-37271137a520-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580498 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580529 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580785 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580785 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.580902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.580884 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.581040 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.581014 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.581254 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.581233 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72508624-36f4-4d4a-992b-37271137a520-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.582894 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.582872 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72508624-36f4-4d4a-992b-37271137a520-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.582977 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.582919 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72508624-36f4-4d4a-992b-37271137a520-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.591070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.591020 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72508624-36f4-4d4a-992b-37271137a520-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.591294 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.591274 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttw5\" (UniqueName: \"kubernetes.io/projected/72508624-36f4-4d4a-992b-37271137a520-kube-api-access-sttw5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nc62c\" (UID: \"72508624-36f4-4d4a-992b-37271137a520\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.668119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.667963 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:01.951029 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.950936 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c"] Nov 02 17:48:01.983787 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:01.983737 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" event={"ID":"72508624-36f4-4d4a-992b-37271137a520","Type":"ContainerStarted","Data":"91dfc2ea4b24b8a19f2e63c0f5e41a4ddfdb80364f20df038d6bfc3b2a780ce0"} Nov 02 17:48:06.808082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:06.808023 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:06.808459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:06.808114 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:06.808459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:06.808144 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:07.012292 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:07.012256 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" event={"ID":"72508624-36f4-4d4a-992b-37271137a520","Type":"ContainerStarted","Data":"c7d0bb80853b0a0f91af16bc03eac5f558d79db7895a4bc0ae86994bcd76b142"} Nov 02 17:48:07.034096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:07.034016 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podStartSLOduration=1.183367799 podStartE2EDuration="6.033998437s" totalImagesPullingTime="4.850630638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:07.032669068 +0000 UTC m=+447.131697265" watchObservedRunningTime="2026-11-02 17:48:07.033998437 +0000 UTC m=+447.133026637" Nov 02 17:48:07.668945 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:07.668912 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:07.670257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:07.670228 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:07.670384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:07.670284 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:08.668658 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:08.668621 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:08.669084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:08.668677 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:09.669088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:09.669027 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:09.669496 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:09.669109 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:10.669220 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:10.669178 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:10.669617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:10.669239 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:11.668647 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:11.668611 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:11.668828 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:11.668664 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:12.668412 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:12.668380 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:12.668875 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:12.668433 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:13.669224 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:13.669192 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 17:48:13.669594 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:13.669244 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 17:48:15.026197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:15.026163 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:15.026680 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:15.026474 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:15.027232 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:15.027210 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" Nov 02 17:48:16.395088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.393350 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d"] Nov 02 17:48:16.397283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.397263 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.413851 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.413817 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d"] Nov 02 17:48:16.430776 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.430746 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.430927 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.430795 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznxb\" (UniqueName: \"kubernetes.io/projected/29129ba4-55d3-48cf-a458-20b8266a7f40-kube-api-access-bznxb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.430927 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.430906 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.431028 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.430933 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.431028 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.430952 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.431028 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.430966 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.431195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.431169 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.431241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.431225 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29129ba4-55d3-48cf-a458-20b8266a7f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.431282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.431264 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532534 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532500 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznxb\" (UniqueName: \"kubernetes.io/projected/29129ba4-55d3-48cf-a458-20b8266a7f40-kube-api-access-bznxb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532566 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532585 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532607 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532625 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532659 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.532725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532694 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29129ba4-55d3-48cf-a458-20b8266a7f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532740 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.532777 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533143 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.533072 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533199 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.533147 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533199 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.533183 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533365 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.533334 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.533542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.533525 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29129ba4-55d3-48cf-a458-20b8266a7f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.535062 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.535026 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.535184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.535169 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.541155 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.541132 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznxb\" (UniqueName: \"kubernetes.io/projected/29129ba4-55d3-48cf-a458-20b8266a7f40-kube-api-access-bznxb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.541269 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.541237 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29129ba4-55d3-48cf-a458-20b8266a7f40-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zk9d\" (UID: \"29129ba4-55d3-48cf-a458-20b8266a7f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.710273 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.710191 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:16.996666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:16.996593 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d"] Nov 02 17:48:17.000026 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:48:17.000000 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29129ba4_55d3_48cf_a458_20b8266a7f40.slice/crio-8601cba810dc44fa09f985b2b5f1ad057841256f680ab3990936098ede055094 WatchSource:0}: Error finding container 8601cba810dc44fa09f985b2b5f1ad057841256f680ab3990936098ede055094: Status 404 returned error can't find the container with id 8601cba810dc44fa09f985b2b5f1ad057841256f680ab3990936098ede055094 Nov 02 17:48:17.002262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:17.002228 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:17.002374 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:17.002310 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:17.002374 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:17.002355 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:48:17.057927 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:17.057896 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" event={"ID":"29129ba4-55d3-48cf-a458-20b8266a7f40","Type":"ContainerStarted","Data":"8601cba810dc44fa09f985b2b5f1ad057841256f680ab3990936098ede055094"} Nov 02 17:48:18.067413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:18.067333 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" event={"ID":"29129ba4-55d3-48cf-a458-20b8266a7f40","Type":"ContainerStarted","Data":"b5ef76b9e7bec8021bca6df7aabafca43538d41baba1d836be0c821a33f112f8"} Nov 02 17:48:18.091142 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:18.091094 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podStartSLOduration=2.091079509 podStartE2EDuration="2.091079509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:18.090251989 +0000 UTC m=+458.189280183" watchObservedRunningTime="2026-11-02 17:48:18.091079509 +0000 UTC m=+458.190107704" Nov 02 17:48:18.711148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:18.711111 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:18.712528 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:18.712504 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 17:48:18.712637 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:18.712555 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 17:48:19.710834 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:19.710801 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 17:48:19.711247 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:19.710855 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 17:48:20.711396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:20.711357 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 17:48:20.711790 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:20.711422 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 17:48:21.711427 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:21.711390 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 17:48:21.711814 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:21.711443 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 17:48:22.711403 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:22.711368 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 17:48:22.711587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:22.711419 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 17:48:23.711229 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:23.711194 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 17:48:23.711440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:23.711271 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 17:48:25.170361 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:25.170322 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:25.170835 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:25.170812 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:25.171577 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:48:25.171557 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" Nov 02 17:48:29.245262 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:48:29.245227 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:48:59.245741 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:48:59.245707 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:49:15.434198 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.434164 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9"] Nov 02 17:49:15.438536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.438510 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.447716 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.447691 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-9dfa27-serving-cert\"" Nov 02 17:49:15.448419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.448399 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9"] Nov 02 17:49:15.562845 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.562813 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.562996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.562937 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsxx\" (UniqueName: \"kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.562996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.562962 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.562996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.562985 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.664462 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.664428 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsxx\" (UniqueName: \"kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.664646 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.664471 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.664646 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.664610 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.664773 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.664744 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.665344 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.665320 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.667078 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.667059 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.667152 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.667096 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.674555 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.674525 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsxx\" (UniqueName: \"kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx\") pod \"maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:15.753494 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:15.753384 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:16.040340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:16.040250 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9"] Nov 02 17:49:16.043832 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:16.043802 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1575541c_bc7c_4a50_a484_ac2f24abde57.slice/crio-27aeb7803fd46d121cc013685c9597d839660c56ac3e471474ed715eab98fe01 WatchSource:0}: Error finding container 27aeb7803fd46d121cc013685c9597d839660c56ac3e471474ed715eab98fe01: Status 404 returned error can't find the container with id 27aeb7803fd46d121cc013685c9597d839660c56ac3e471474ed715eab98fe01 Nov 02 17:49:16.331412 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:16.331359 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" event={"ID":"1575541c-bc7c-4a50-a484-ac2f24abde57","Type":"ContainerStarted","Data":"27aeb7803fd46d121cc013685c9597d839660c56ac3e471474ed715eab98fe01"} Nov 02 17:49:17.337990 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:17.337956 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" event={"ID":"1575541c-bc7c-4a50-a484-ac2f24abde57","Type":"ContainerStarted","Data":"5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08"} Nov 02 17:49:17.338503 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:17.338086 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:17.362556 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:17.362505 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" podStartSLOduration=1.942706198 podStartE2EDuration="2.362489207s" totalImagesPullingTime="419.783009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:17.360792738 +0000 UTC m=+517.459820937" watchObservedRunningTime="2026-11-02 17:49:17.362489207 +0000 UTC m=+517.461517403" Nov 02 17:49:18.303766 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.303734 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw"] Nov 02 17:49:18.307409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.307387 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.323496 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.323472 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw"] Nov 02 17:49:18.342673 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.342636 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ea752f-serving-cert\"" Nov 02 17:49:18.393210 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.393181 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.393351 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.393227 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.393402 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.393381 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbz88\" (UniqueName: \"kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.393458 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.393443 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.495122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.495081 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbz88\" (UniqueName: \"kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.495303 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.495161 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.495303 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.495274 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.495429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.495313 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.496016 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.495986 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.497984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.497964 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.498096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.497984 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.505950 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.505916 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbz88\" (UniqueName: \"kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88\") pod \"maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.622184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.622161 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:18.897999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:18.897954 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw"] Nov 02 17:49:18.900143 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:18.900102 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0dbad12_dc7f_4931_afd7_50492a5d8358.slice/crio-229032452a83b16b9dc26e5235f1a8f54a95d06c3e80c45eb2385699b451b6f5 WatchSource:0}: Error finding container 229032452a83b16b9dc26e5235f1a8f54a95d06c3e80c45eb2385699b451b6f5: Status 404 returned error can't find the container with id 229032452a83b16b9dc26e5235f1a8f54a95d06c3e80c45eb2385699b451b6f5 Nov 02 17:49:19.349260 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:19.349228 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" event={"ID":"c0dbad12-dc7f-4931-afd7-50492a5d8358","Type":"ContainerStarted","Data":"229032452a83b16b9dc26e5235f1a8f54a95d06c3e80c45eb2385699b451b6f5"} Nov 02 17:49:20.355496 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.355459 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" event={"ID":"c0dbad12-dc7f-4931-afd7-50492a5d8358","Type":"ContainerStarted","Data":"a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f"} Nov 02 17:49:20.355904 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.355568 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:20.412236 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.412169 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" podStartSLOduration=2.008096323 podStartE2EDuration="2.412150603s" totalImagesPullingTime="404.05428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.411944721 +0000 UTC m=+520.510972920" watchObservedRunningTime="2026-11-02 17:49:20.412150603 +0000 UTC m=+520.511178798" Nov 02 17:49:20.438353 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.438326 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p"] Nov 02 17:49:20.441945 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.441926 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.468576 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.468545 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p"] Nov 02 17:49:20.506214 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.506180 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ea3867\"" Nov 02 17:49:20.512614 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.512571 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-695xt\" (UniqueName: \"kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt\") pod \"payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.512762 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.512630 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.514483 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.514470 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ea3867-dockercfg-s6rrb\"" Nov 02 17:49:20.613671 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.613567 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695xt\" (UniqueName: \"kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt\") pod \"payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.613671 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.613619 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.614386 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.614358 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.626553 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.626528 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-695xt\" (UniqueName: \"kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt\") pod \"payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:20.758471 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:20.758426 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:21.045450 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:21.045422 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p"] Nov 02 17:49:21.047937 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:21.047909 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b7f858_3475_4ba6_ba7e_ac334712e015.slice/crio-321aa48642aefc6174cea5ecdbd9c245b984ff4b01fc76c0f58a455bf70441c5 WatchSource:0}: Error finding container 321aa48642aefc6174cea5ecdbd9c245b984ff4b01fc76c0f58a455bf70441c5: Status 404 returned error can't find the container with id 321aa48642aefc6174cea5ecdbd9c245b984ff4b01fc76c0f58a455bf70441c5 Nov 02 17:49:21.361232 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:21.361191 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" event={"ID":"06b7f858-3475-4ba6-ba7e-ac334712e015","Type":"ContainerStarted","Data":"321aa48642aefc6174cea5ecdbd9c245b984ff4b01fc76c0f58a455bf70441c5"} Nov 02 17:49:23.349287 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:23.349260 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:49:23.374146 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:23.374042 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" event={"ID":"06b7f858-3475-4ba6-ba7e-ac334712e015","Type":"ContainerStarted","Data":"67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0"} Nov 02 17:49:23.374300 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:23.374195 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:23.392487 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:23.392421 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" podStartSLOduration=1.321087414 podStartE2EDuration="3.392402271s" totalImagesPullingTime="2.071314857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:23.39184645 +0000 UTC m=+523.490874644" watchObservedRunningTime="2026-11-02 17:49:23.392402271 +0000 UTC m=+523.491430466" Nov 02 17:49:26.367419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:26.367387 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 17:49:29.246739 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:49:29.246700 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:49:34.382703 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:34.382671 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:49:37.081821 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.081785 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v"] Nov 02 17:49:37.088329 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.088307 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.097697 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.097667 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"openshift-service-ca.crt\"" Nov 02 17:49:37.097864 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.097709 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"kube-root-ca.crt\"" Nov 02 17:49:37.098400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.098371 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v"] Nov 02 17:49:37.105203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.105184 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"simulator-w2-bb4ecb-kserve-self-signed-certs\"" Nov 02 17:49:37.105307 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.105210 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"default-dockercfg-gjl4f\"" Nov 02 17:49:37.178071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.178014 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.178253 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.178183 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mml\" (UniqueName: \"kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.178336 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.178311 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.178429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.178412 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.178488 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.178464 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.178537 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.178488 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279235 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279203 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279418 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279262 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279418 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279288 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279418 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279333 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279418 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279354 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mml\" (UniqueName: \"kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279418 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279390 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279664 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279813 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279792 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.279867 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.279796 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.281486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.281461 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.282132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.282113 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.289756 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.289736 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mml\" (UniqueName: \"kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml\") pod \"simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.402602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.402507 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:37.703090 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:37.702989 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v"] Nov 02 17:49:37.706073 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:37.706027 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6eadac3_d485_445e_8edd_107faff8c296.slice/crio-1a1094a0ef7fee89d184932a088b2d4a4fd2ae43aa1851985437c404e0414715 WatchSource:0}: Error finding container 1a1094a0ef7fee89d184932a088b2d4a4fd2ae43aa1851985437c404e0414715: Status 404 returned error can't find the container with id 1a1094a0ef7fee89d184932a088b2d4a4fd2ae43aa1851985437c404e0414715 Nov 02 17:49:38.455322 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:38.455187 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" event={"ID":"f6eadac3-d485-445e-8edd-107faff8c296","Type":"ContainerStarted","Data":"1a1094a0ef7fee89d184932a088b2d4a4fd2ae43aa1851985437c404e0414715"} Nov 02 17:49:39.461919 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.461882 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" event={"ID":"f6eadac3-d485-445e-8edd-107faff8c296","Type":"ContainerStarted","Data":"ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84"} Nov 02 17:49:39.482964 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.482914 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podStartSLOduration=1.280683835 podStartE2EDuration="2.482901032s" totalImagesPullingTime="1.202217197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:39.481365037 +0000 UTC m=+539.580393236" watchObservedRunningTime="2026-11-02 17:49:39.482901032 +0000 UTC m=+539.581929227" Nov 02 17:49:39.818526 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.818448 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw"] Nov 02 17:49:39.822879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.822854 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:39.828611 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.828588 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"default-dockercfg-jptzs\"" Nov 02 17:49:39.830544 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.830518 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw"] Nov 02 17:49:39.831173 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.831153 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"simulator-w6-7b0683-kserve-self-signed-certs\"" Nov 02 17:49:39.831471 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.831451 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"kube-root-ca.crt\"" Nov 02 17:49:39.831560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:39.831499 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"openshift-service-ca.crt\"" Nov 02 17:49:40.009000 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.008970 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.009182 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.009016 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.009182 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.009035 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.009182 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.009173 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.009287 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.009194 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.009287 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.009232 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz5v\" (UniqueName: \"kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.110834 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.110795 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.110863 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.110892 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.110933 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.110956 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111265 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.111023 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz5v\" (UniqueName: \"kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111338 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.111268 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111539 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.111519 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.111602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.111586 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.113120 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.113094 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.113901 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.113885 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"simulator-w6-7b0683-kserve-self-signed-certs\"" Nov 02 17:49:40.119542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.119524 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"kube-root-ca.crt\"" Nov 02 17:49:40.123724 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.123701 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.130059 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.130032 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"openshift-service-ca.crt\"" Nov 02 17:49:40.140160 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.140139 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz5v\" (UniqueName: \"kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v\") pod \"simulator-w6-7b0683-kserve-85d9954979-rvzpw\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.403175 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.403103 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:40.440517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.440481 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"default-dockercfg-jptzs\"" Nov 02 17:49:40.447863 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.447834 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:40.727400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.727375 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw"] Nov 02 17:49:40.729905 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:40.729877 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd61ad35_435a_4acd_bb1b_9dd8f09e5cd5.slice/crio-fc4a94ee118acc236d0fe0941f30d2adf172ac023cd95e1d8a05a5c3e32405f3 WatchSource:0}: Error finding container fc4a94ee118acc236d0fe0941f30d2adf172ac023cd95e1d8a05a5c3e32405f3: Status 404 returned error can't find the container with id fc4a94ee118acc236d0fe0941f30d2adf172ac023cd95e1d8a05a5c3e32405f3 Nov 02 17:49:40.987477 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.987406 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2"] Nov 02 17:49:40.995379 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.995355 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.000142 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:40.999897 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2"] Nov 02 17:49:41.001233 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.001207 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"openshift-service-ca.crt\"" Nov 02 17:49:41.001352 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.001235 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"kube-root-ca.crt\"" Nov 02 17:49:41.002213 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.002174 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"default-dockercfg-mb25d\"" Nov 02 17:49:41.002330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.002225 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"simulator-w5-ea3867-kserve-self-signed-certs\"" Nov 02 17:49:41.119295 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.119263 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzrj\" (UniqueName: \"kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.119499 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.119337 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.119499 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.119419 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.119499 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.119462 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.119662 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.119510 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.119712 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.119697 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221083 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221030 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221162 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzrj\" (UniqueName: \"kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221210 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221224 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221241 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221453 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221280 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221675 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221639 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221875 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221819 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.221875 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.221821 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.223537 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.223509 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.223824 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.223802 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.229720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.229695 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzrj\" (UniqueName: \"kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj\") pod \"simulator-w5-ea3867-kserve-769c76b546-pfgd2\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.310296 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.310217 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:41.474620 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.474568 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" event={"ID":"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5","Type":"ContainerStarted","Data":"72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94"} Nov 02 17:49:41.474815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.474630 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" event={"ID":"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5","Type":"ContainerStarted","Data":"fc4a94ee118acc236d0fe0941f30d2adf172ac023cd95e1d8a05a5c3e32405f3"} Nov 02 17:49:41.492005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.491959 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podStartSLOduration=2.491944421 podStartE2EDuration="2.491944421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:41.491228516 +0000 UTC m=+541.590256737" watchObservedRunningTime="2026-11-02 17:49:41.491944421 +0000 UTC m=+541.590972616" Nov 02 17:49:41.589713 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:41.589684 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2"] Nov 02 17:49:41.591961 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:41.591930 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfdf4b07_d0e8_4f27_8c2d_f546992b6f5a.slice/crio-879db3afd0f7164af5bef9c809e6ca6adf61c17b2264c4081b42954d9b1e70c0 WatchSource:0}: Error finding container 879db3afd0f7164af5bef9c809e6ca6adf61c17b2264c4081b42954d9b1e70c0: Status 404 returned error can't find the container with id 879db3afd0f7164af5bef9c809e6ca6adf61c17b2264c4081b42954d9b1e70c0 Nov 02 17:49:42.412536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.412459 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t"] Nov 02 17:49:42.419137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.419033 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.425404 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.424776 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t"] Nov 02 17:49:42.425594 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.425230 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"simulator-w4-ea752f-kserve-self-signed-certs\"" Nov 02 17:49:42.428569 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.428536 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"default-dockercfg-wm8ft\"" Nov 02 17:49:42.428721 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.428620 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"kube-root-ca.crt\"" Nov 02 17:49:42.429422 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.429117 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"openshift-service-ca.crt\"" Nov 02 17:49:42.429859 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.429835 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.429929 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.429886 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.429995 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.429976 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.430075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.430014 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqpp\" (UniqueName: \"kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.430144 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.430102 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.430200 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.430145 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.448512 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.448481 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:42.482246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.482210 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" event={"ID":"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a","Type":"ContainerStarted","Data":"9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3"} Nov 02 17:49:42.482389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.482256 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" event={"ID":"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a","Type":"ContainerStarted","Data":"879db3afd0f7164af5bef9c809e6ca6adf61c17b2264c4081b42954d9b1e70c0"} Nov 02 17:49:42.506202 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.506140 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podStartSLOduration=2.506123729 podStartE2EDuration="2.506123729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:42.503709447 +0000 UTC m=+542.602737644" watchObservedRunningTime="2026-11-02 17:49:42.506123729 +0000 UTC m=+542.605151924" Nov 02 17:49:42.531487 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531460 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.531621 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531499 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.531687 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531661 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.531744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531715 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.531794 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531751 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.531794 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531779 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqpp\" (UniqueName: \"kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.531936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531917 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.532000 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.531946 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.532612 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.532586 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.534093 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.534061 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.534611 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.534588 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.539872 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.539847 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqpp\" (UniqueName: \"kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp\") pod \"simulator-w4-ea752f-kserve-86cf5c445-t667t\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:42.734978 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:42.734893 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:43.014175 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:43.014152 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t"] Nov 02 17:49:43.016033 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:43.016009 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e6c74f3_c984_4f3e_90f7_525c66ee8a2a.slice/crio-31f99a369a03d585fa03375428aa4d69580d66fb25b4b4aa6a1feabee15c9bad WatchSource:0}: Error finding container 31f99a369a03d585fa03375428aa4d69580d66fb25b4b4aa6a1feabee15c9bad: Status 404 returned error can't find the container with id 31f99a369a03d585fa03375428aa4d69580d66fb25b4b4aa6a1feabee15c9bad Nov 02 17:49:43.311350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:43.311247 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:43.488262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:43.488226 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" event={"ID":"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a","Type":"ContainerStarted","Data":"f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d"} Nov 02 17:49:43.488262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:43.488268 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" event={"ID":"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a","Type":"ContainerStarted","Data":"31f99a369a03d585fa03375428aa4d69580d66fb25b4b4aa6a1feabee15c9bad"} Nov 02 17:49:43.508944 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:43.508892 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podStartSLOduration=1.5088776080000001 podStartE2EDuration="1.508877608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:43.507565857 +0000 UTC m=+543.606594066" watchObservedRunningTime="2026-11-02 17:49:43.508877608 +0000 UTC m=+543.607905802" Nov 02 17:49:43.736120 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:43.736079 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:47.403164 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:47.403129 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:47.416478 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:47.416449 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:47.526633 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:47.526602 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:49:50.448980 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:50.448904 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:50.461932 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:50.461906 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:50.537097 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:50.537067 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:49:51.311012 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.310974 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:51.324601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.324570 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:51.541996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.541970 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:49:51.918736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.918704 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r"] Nov 02 17:49:51.922540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.922516 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:51.933227 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.933199 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"default-dockercfg-cmqbw\"" Nov 02 17:49:51.934113 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.934088 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"kube-root-ca.crt\"" Nov 02 17:49:51.934267 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.934126 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"simulator-w3-2a295c-kserve-self-signed-certs\"" Nov 02 17:49:51.934267 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.934142 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"openshift-service-ca.crt\"" Nov 02 17:49:51.935084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:51.935060 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r"] Nov 02 17:49:52.022752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.022720 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.022951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.022769 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhcj\" (UniqueName: \"kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.022951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.022787 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.022951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.022902 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.023164 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.023001 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.023164 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.023135 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.123912 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.123867 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.123948 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhcj\" (UniqueName: \"kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.123970 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.123986 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.124027 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124394 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.124164 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.124407 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124571 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.124548 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.124641 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.124622 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.126399 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.126374 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.126863 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.126838 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.132319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.132292 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhcj\" (UniqueName: \"kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj\") pod \"simulator-w3-2a295c-kserve-7c6f6d676b-czd4r\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.236410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.236335 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:52.521606 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.521534 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r"] Nov 02 17:49:52.524828 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:49:52.524800 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod631fdc58_d4ce_495b_87e1_342fc876639d.slice/crio-584bf3b6a9960f5a7878d518567279fd909507932033fb753a6416ce60255b8a WatchSource:0}: Error finding container 584bf3b6a9960f5a7878d518567279fd909507932033fb753a6416ce60255b8a: Status 404 returned error can't find the container with id 584bf3b6a9960f5a7878d518567279fd909507932033fb753a6416ce60255b8a Nov 02 17:49:52.537848 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.537784 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" event={"ID":"631fdc58-d4ce-495b-87e1-342fc876639d","Type":"ContainerStarted","Data":"584bf3b6a9960f5a7878d518567279fd909507932033fb753a6416ce60255b8a"} Nov 02 17:49:52.735403 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.735367 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:52.748298 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:52.748271 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:53.543832 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:53.543799 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" event={"ID":"631fdc58-d4ce-495b-87e1-342fc876639d","Type":"ContainerStarted","Data":"a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038"} Nov 02 17:49:53.555854 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:53.555823 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:49:53.567236 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:53.567178 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podStartSLOduration=2.567159078 podStartE2EDuration="2.567159078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:53.563304681 +0000 UTC m=+553.662332878" watchObservedRunningTime="2026-11-02 17:49:53.567159078 +0000 UTC m=+553.666187274" Nov 02 17:49:54.237328 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:49:54.237288 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:49:59.246379 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:49:59.246338 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:50:02.236939 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.236902 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:50:02.250596 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.250564 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:50:02.605166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.605131 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:50:02.851656 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.851615 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d"] Nov 02 17:50:02.856084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.856003 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:02.863320 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.863294 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9dfa27\"/\"premium-w1-9dfa27-kserve-self-signed-certs\"" Nov 02 17:50:02.863472 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.863367 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9dfa27\"/\"default-dockercfg-kg7p6\"" Nov 02 17:50:02.863472 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.863411 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9dfa27\"/\"openshift-service-ca.crt\"" Nov 02 17:50:02.863585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.863491 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9dfa27\"/\"kube-root-ca.crt\"" Nov 02 17:50:02.865930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.865906 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d"] Nov 02 17:50:02.937578 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.937541 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:02.937790 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.937596 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:02.937790 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.937688 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:02.937790 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.937758 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:02.937957 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.937805 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwvn\" (UniqueName: \"kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:02.938016 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:02.937956 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039121 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039081 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039327 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039162 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwvn\" (UniqueName: \"kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039327 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039267 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039348 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039399 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039421 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039853 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039821 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039821 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.039987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.039905 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.041947 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.041877 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.042205 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.042163 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.050914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.050886 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwvn\" (UniqueName: \"kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn\") pod \"premium-w1-9dfa27-kserve-74d999b4b4-8ck8d\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.171188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.171094 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:03.450443 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.450415 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d"] Nov 02 17:50:03.453426 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:50:03.453390 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a32b109_a7d9_4804_a76e_028a78fbffb1.slice/crio-fc4e416ee806f9c24995cda07c4254ad546f839911c394e188fffc1b609a711f WatchSource:0}: Error finding container fc4e416ee806f9c24995cda07c4254ad546f839911c394e188fffc1b609a711f: Status 404 returned error can't find the container with id fc4e416ee806f9c24995cda07c4254ad546f839911c394e188fffc1b609a711f Nov 02 17:50:03.596075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.596023 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" event={"ID":"3a32b109-a7d9-4804-a76e-028a78fbffb1","Type":"ContainerStarted","Data":"621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee"} Nov 02 17:50:03.596075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.596091 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" event={"ID":"3a32b109-a7d9-4804-a76e-028a78fbffb1","Type":"ContainerStarted","Data":"fc4e416ee806f9c24995cda07c4254ad546f839911c394e188fffc1b609a711f"} Nov 02 17:50:03.615953 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:03.615902 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podStartSLOduration=1.6158798220000001 podStartE2EDuration="1.615879822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:03.61492663 +0000 UTC m=+563.713954837" watchObservedRunningTime="2026-11-02 17:50:03.615879822 +0000 UTC m=+563.714908069" Nov 02 17:50:04.171427 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:04.171372 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:12.498613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.498576 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf"] Nov 02 17:50:12.502460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.502442 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.506511 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.506488 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"premium-w5-ea3867-kserve-self-signed-certs\"" Nov 02 17:50:12.511587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.511564 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf"] Nov 02 17:50:12.639321 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.639282 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.639506 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.639384 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kv7\" (UniqueName: \"kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.639506 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.639445 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.639634 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.639547 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.639634 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.639599 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.639807 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.639790 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741234 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741200 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741398 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741278 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741398 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741328 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kv7\" (UniqueName: \"kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741398 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741361 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741398 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741392 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741595 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741425 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741681 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741646 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.741742 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741702 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.742002 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.741979 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.743690 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.743663 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.743858 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.743843 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.750654 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.750599 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kv7\" (UniqueName: \"kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7\") pod \"premium-w5-ea3867-kserve-6f444bb759-4tdsf\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:12.819001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:12.818971 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:13.090819 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.090792 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf"] Nov 02 17:50:13.093022 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:50:13.092993 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1da72f_9b81_46ea_96d9_4f5cd4732415.slice/crio-0e0390f45681a68be4a9d5f1810a00b9bae1d884e45b6b19b7adcc3271c70bfb WatchSource:0}: Error finding container 0e0390f45681a68be4a9d5f1810a00b9bae1d884e45b6b19b7adcc3271c70bfb: Status 404 returned error can't find the container with id 0e0390f45681a68be4a9d5f1810a00b9bae1d884e45b6b19b7adcc3271c70bfb Nov 02 17:50:13.172105 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.172076 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:13.184761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.184739 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:13.652283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.652250 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" event={"ID":"3d1da72f-9b81-46ea-96d9-4f5cd4732415","Type":"ContainerStarted","Data":"ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163"} Nov 02 17:50:13.652722 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.652292 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" event={"ID":"3d1da72f-9b81-46ea-96d9-4f5cd4732415","Type":"ContainerStarted","Data":"0e0390f45681a68be4a9d5f1810a00b9bae1d884e45b6b19b7adcc3271c70bfb"} Nov 02 17:50:13.663706 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.663684 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:50:13.671348 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.671307 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podStartSLOduration=1.6712929330000001 podStartE2EDuration="1.671292933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:13.668721844 +0000 UTC m=+573.767750041" watchObservedRunningTime="2026-11-02 17:50:13.671292933 +0000 UTC m=+573.770321127" Nov 02 17:50:13.819395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:13.819362 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:19.136104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.136073 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2"] Nov 02 17:50:19.140953 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.140933 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.149423 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.149401 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"premium-w3-2a295c-kserve-self-signed-certs\"" Nov 02 17:50:19.151821 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.151794 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2"] Nov 02 17:50:19.209144 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.209110 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.209286 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.209230 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.209286 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.209272 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqrv\" (UniqueName: \"kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.209359 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.209331 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.209397 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.209360 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.209431 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.209394 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.310668 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.310635 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.310880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.310675 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqrv\" (UniqueName: \"kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.310880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.310699 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.310880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.310737 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.310880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.310783 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.311118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.310835 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.311258 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.311231 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.311397 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.311372 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.311468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.311388 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.313091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.313043 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.313468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.313449 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.318553 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.318532 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqrv\" (UniqueName: \"kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv\") pod \"premium-w3-2a295c-kserve-6c744f99b7-ltkv2\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.459368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.459285 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:19.734785 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:19.734757 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2"] Nov 02 17:50:19.736223 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:50:19.736194 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95061dfc_67fd_434b_9ace_831026281d78.slice/crio-40d2e23268d701999313ca487b4b99b7da1477aebaf0234bbb3b7ee3197df77d WatchSource:0}: Error finding container 40d2e23268d701999313ca487b4b99b7da1477aebaf0234bbb3b7ee3197df77d: Status 404 returned error can't find the container with id 40d2e23268d701999313ca487b4b99b7da1477aebaf0234bbb3b7ee3197df77d Nov 02 17:50:20.690383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:20.690335 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" event={"ID":"95061dfc-67fd-434b-9ace-831026281d78","Type":"ContainerStarted","Data":"d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9"} Nov 02 17:50:20.690861 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:20.690391 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" event={"ID":"95061dfc-67fd-434b-9ace-831026281d78","Type":"ContainerStarted","Data":"40d2e23268d701999313ca487b4b99b7da1477aebaf0234bbb3b7ee3197df77d"} Nov 02 17:50:20.708836 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:20.708765 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podStartSLOduration=1.70875294 podStartE2EDuration="1.70875294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:20.707696661 +0000 UTC m=+580.806724889" watchObservedRunningTime="2026-11-02 17:50:20.70875294 +0000 UTC m=+580.807781168" Nov 02 17:50:21.459875 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:21.459842 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:22.819589 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:22.819545 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:22.832193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:22.832167 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:23.717911 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:23.717883 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:50:29.246497 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:50:29.246459 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:50:29.459581 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:29.459543 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:29.472396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:29.472367 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:29.750509 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:29.750482 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:50:37.301137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.301096 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l"] Nov 02 17:50:37.304956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.304933 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.322498 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.322465 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ea3867-openshift-default-dockercfg-sdtfb\"" Nov 02 17:50:37.324643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.324612 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l"] Nov 02 17:50:37.406857 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.406819 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.406867 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.406896 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.407003 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.407070 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.407144 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.407183 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407352 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.407209 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.407352 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.407287 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.508522 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508490 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.508763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508558 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.508763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508604 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.508763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508644 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.508763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508676 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.508763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508704 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509037 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508778 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509037 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508816 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509037 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.508843 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509210 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.509086 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509435 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.509408 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.509447 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509647 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.509622 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.509947 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.509922 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.511800 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.511774 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.512021 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.512003 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.522111 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.522078 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.522517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.522499 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.618907 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.618866 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:37.907330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.907296 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l"] Nov 02 17:50:37.909412 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:50:37.909386 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ba9bc0_8708_4e4b_854d_c2eb5256260e.slice/crio-a5d3e7de22d84f42fb69b90938529f7edd6970450ecb880d95029021bbfb0ade WatchSource:0}: Error finding container a5d3e7de22d84f42fb69b90938529f7edd6970450ecb880d95029021bbfb0ade: Status 404 returned error can't find the container with id a5d3e7de22d84f42fb69b90938529f7edd6970450ecb880d95029021bbfb0ade Nov 02 17:50:37.912212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.912171 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:37.912300 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.912273 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:37.912339 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:37.912322 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:38.784952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:38.784905 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" event={"ID":"68ba9bc0-8708-4e4b-854d-c2eb5256260e","Type":"ContainerStarted","Data":"4ffb9f2420a97512199d5b45dac0b8636fc4d054bef9ac0eeba9e307d667eb61"} Nov 02 17:50:38.785357 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:38.784975 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" event={"ID":"68ba9bc0-8708-4e4b-854d-c2eb5256260e","Type":"ContainerStarted","Data":"a5d3e7de22d84f42fb69b90938529f7edd6970450ecb880d95029021bbfb0ade"} Nov 02 17:50:38.806985 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:38.806926 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podStartSLOduration=1.80690766 podStartE2EDuration="1.80690766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:38.80538485 +0000 UTC m=+598.904413047" watchObservedRunningTime="2026-11-02 17:50:38.80690766 +0000 UTC m=+598.905935855" Nov 02 17:50:39.619244 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:39.619201 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:39.620657 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:39.620626 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 17:50:39.620779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:39.620677 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 17:50:40.108905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:40.108876 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:50:40.109337 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:40.109003 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:50:40.619171 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:40.619140 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 17:50:40.619359 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:40.619185 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 17:50:41.619765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:41.619728 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 17:50:41.620185 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:41.619780 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 17:50:42.619210 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:42.619174 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 17:50:42.619380 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:42.619227 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 17:50:43.619540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:43.619504 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 17:50:43.620036 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:43.619567 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 17:50:44.625132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:44.625100 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:44.625538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:44.625398 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:44.625959 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:44.625943 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:50:51.720095 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.720034 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp"] Nov 02 17:50:51.723501 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.723483 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.728336 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.728316 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f37d16-openshift-default-dockercfg-5tcm2\"" Nov 02 17:50:51.734122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.734099 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp"] Nov 02 17:50:51.782713 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782668 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.782877 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782723 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.782877 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782795 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.782877 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782869 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.783014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782901 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.783014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782966 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.783014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.782995 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.783230 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.783041 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtwt\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.783230 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.783195 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884320 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884278 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884367 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtwt\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884412 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884438 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884480 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884506 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884578 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884623 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884659 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.884955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.884874 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.885107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.885088 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.885196 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.885130 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.885391 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.885369 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.885459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.885392 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.886616 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.886594 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.886937 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.886918 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.892914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.892892 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:51.893214 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:51.893197 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtwt\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:52.039137 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.039062 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:52.069943 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.069907 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv"] Nov 02 17:50:52.073964 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.073944 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.081829 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.081792 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb4ecb-openshift-default-dockercfg-j9575\"" Nov 02 17:50:52.085787 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.084584 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv"] Nov 02 17:50:52.187797 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.187763 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.187951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.187819 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.187951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.187903 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.188034 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.188008 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.188114 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.188099 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.188155 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.188121 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.188155 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.188139 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpd45\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.188230 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.188197 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.188230 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.188220 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289532 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289561 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289581 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpd45\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289601 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289627 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289664 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289727 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289765 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.289905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.289860 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.290248 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.290040 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.290248 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.290213 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.290319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.290292 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.290359 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.290295 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.290467 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.290437 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.292106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.292082 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.292260 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.292245 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.298435 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.298417 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.298525 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.298506 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpd45\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.326464 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.326434 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp"] Nov 02 17:50:52.328755 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:50:52.328736 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9a14e5_f0a7_44fe_ad6c_7c2fbe169e6a.slice/crio-f7be0a42971254cb8215e1c672999d9f480f8b52a0d5a5bbf17db6090b36f6e0 WatchSource:0}: Error finding container f7be0a42971254cb8215e1c672999d9f480f8b52a0d5a5bbf17db6090b36f6e0: Status 404 returned error can't find the container with id f7be0a42971254cb8215e1c672999d9f480f8b52a0d5a5bbf17db6090b36f6e0 Nov 02 17:50:52.331082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.331024 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:52.331158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.331132 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:52.331197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.331161 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:52.391388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.391370 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:52.668109 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.668081 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv"] Nov 02 17:50:52.671104 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:50:52.671072 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fc7ad3_63dd_4834_9ef9_3a7b20cbec7a.slice/crio-972c205e746a816125a8889887effc57a6346ae0453e6d20f2d96b41d0c98eb8 WatchSource:0}: Error finding container 972c205e746a816125a8889887effc57a6346ae0453e6d20f2d96b41d0c98eb8: Status 404 returned error can't find the container with id 972c205e746a816125a8889887effc57a6346ae0453e6d20f2d96b41d0c98eb8 Nov 02 17:50:52.673547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.673514 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:52.673638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.673585 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:52.673638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.673614 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:50:52.858728 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.858688 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" event={"ID":"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a","Type":"ContainerStarted","Data":"c00438e99f4ebf4bee55f3d0e2f8ee050b05643103912697a77bf9c97854ca2f"} Nov 02 17:50:52.859184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.858759 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" event={"ID":"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a","Type":"ContainerStarted","Data":"972c205e746a816125a8889887effc57a6346ae0453e6d20f2d96b41d0c98eb8"} Nov 02 17:50:52.860340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.860310 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" event={"ID":"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a","Type":"ContainerStarted","Data":"482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7"} Nov 02 17:50:52.860447 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.860374 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" event={"ID":"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a","Type":"ContainerStarted","Data":"f7be0a42971254cb8215e1c672999d9f480f8b52a0d5a5bbf17db6090b36f6e0"} Nov 02 17:50:52.880022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.879976 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podStartSLOduration=0.879961742 podStartE2EDuration="879.961742ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:52.878827639 +0000 UTC m=+612.977855837" watchObservedRunningTime="2026-11-02 17:50:52.879961742 +0000 UTC m=+612.978989937" Nov 02 17:50:52.899379 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:52.899334 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podStartSLOduration=1.899319067 podStartE2EDuration="1.899319067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:52.89812723 +0000 UTC m=+612.997155426" watchObservedRunningTime="2026-11-02 17:50:52.899319067 +0000 UTC m=+612.998347263" Nov 02 17:50:53.039614 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:53.039527 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:50:53.041140 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:53.041102 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:53.041270 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:53.041186 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:53.394073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:53.392268 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:50:53.394073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:53.393728 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:53.394073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:53.393779 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:50:54.058071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:54.048168 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:54.058071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:54.048231 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:54.392492 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:54.392456 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:54.392677 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:54.392505 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:50:55.044069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:55.040392 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:55.044069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:55.040460 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:55.393484 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:55.393442 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:55.394090 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:55.394035 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:50:56.059073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:56.058479 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:56.059073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:56.058543 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:56.392061 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:56.392016 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:56.392234 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:56.392082 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:50:57.040534 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:57.040491 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:57.041180 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:57.041146 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:57.392244 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:57.392202 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:57.392429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:57.392260 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:50:58.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:58.040459 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:58.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:58.040528 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:58.392551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:58.392515 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:58.392734 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:58.392578 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:50:59.040413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:59.040382 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 17:50:59.040617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:59.040435 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 17:50:59.248531 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:50:59.248485 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:50:59.391915 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:59.391870 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 17:50:59.392113 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:50:59.391948 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 17:51:00.044181 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:00.044150 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:51:00.044547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:00.044525 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:51:00.045106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:00.045090 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:51:00.408897 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:00.408865 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:51:00.409332 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:00.409144 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:51:00.409907 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:00.409885 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 17:51:22.222774 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.222701 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv"] Nov 02 17:51:22.226425 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.226403 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.230586 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.230563 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2a295c-openshift-default-dockercfg-kqbfr\"" Nov 02 17:51:22.237080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.237035 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv"] Nov 02 17:51:22.370986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.370957 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371147 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.370996 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnbm\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371147 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371015 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371147 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371095 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371147 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371125 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371280 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371242 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371280 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371275 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371343 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371297 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.371429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.371409 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472399 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472365 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472574 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472408 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472574 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472467 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472574 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472489 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472574 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472507 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472590 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472631 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472661 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnbm\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.472769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.472687 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.473328 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.473257 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.473530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.473502 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.473610 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.473542 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.473649 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.473616 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.473731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.473706 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.475768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.475739 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.476014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.475994 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.483197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.483162 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.483536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.483517 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnbm\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.517681 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.517651 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5"] Nov 02 17:51:22.522713 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.522681 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.526803 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.526782 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7b0683-openshift-default-dockercfg-rp64v\"" Nov 02 17:51:22.537458 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.537426 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5"] Nov 02 17:51:22.540648 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.540614 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:22.674937 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.674900 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bqm\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.674968 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675011 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675041 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675284 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675171 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675284 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675203 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675284 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675266 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675379 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675327 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.675379 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.675352 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.776827 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.776743 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.776827 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.776783 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.776830 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bqm\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.776869 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.776884 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.776904 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777118 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777164 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777228 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777288 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777487 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777614 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777564 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777614 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777597 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.777714 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.777607 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.779267 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.779246 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.779872 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.779851 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.785191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.785170 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.785278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.785233 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bqm\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:22.831490 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.831463 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv"] Nov 02 17:51:22.833092 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:51:22.833064 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21914c59_5090_4c83_8605_f4801a52d8d0.slice/crio-588cfbd94a16d12a02a1427790640540ae1effbf2cb15be45a443a13fd850bde WatchSource:0}: Error finding container 588cfbd94a16d12a02a1427790640540ae1effbf2cb15be45a443a13fd850bde: Status 404 returned error can't find the container with id 588cfbd94a16d12a02a1427790640540ae1effbf2cb15be45a443a13fd850bde Nov 02 17:51:22.835397 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.835365 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:22.835479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.835426 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:22.835479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.835458 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:22.839895 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:22.839876 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:23.042035 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.041951 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" event={"ID":"21914c59-5090-4c83-8605-f4801a52d8d0","Type":"ContainerStarted","Data":"4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6"} Nov 02 17:51:23.042035 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.042005 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" event={"ID":"21914c59-5090-4c83-8605-f4801a52d8d0","Type":"ContainerStarted","Data":"588cfbd94a16d12a02a1427790640540ae1effbf2cb15be45a443a13fd850bde"} Nov 02 17:51:23.070738 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.070656 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podStartSLOduration=1.070641296 podStartE2EDuration="1.070641296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:23.06925385 +0000 UTC m=+643.168282046" watchObservedRunningTime="2026-11-02 17:51:23.070641296 +0000 UTC m=+643.169669490" Nov 02 17:51:23.127999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.127971 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5"] Nov 02 17:51:23.130489 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:51:23.130455 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7f7261_02e4_40a3_b507_112ce6ce4da4.slice/crio-ee81946ee3c59c4bf7bac307a6d8b712be32e405bb1352649b713ec38e7415b5 WatchSource:0}: Error finding container ee81946ee3c59c4bf7bac307a6d8b712be32e405bb1352649b713ec38e7415b5: Status 404 returned error can't find the container with id ee81946ee3c59c4bf7bac307a6d8b712be32e405bb1352649b713ec38e7415b5 Nov 02 17:51:23.133272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.133241 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:23.133408 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.133309 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:23.133408 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.133339 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:23.564075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.561454 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:23.564712 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.564681 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:23.564890 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:23.564862 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:24.059602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.059490 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" event={"ID":"1e7f7261-02e4-40a3-b507-112ce6ce4da4","Type":"ContainerStarted","Data":"51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb"} Nov 02 17:51:24.059602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.059563 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" event={"ID":"1e7f7261-02e4-40a3-b507-112ce6ce4da4","Type":"ContainerStarted","Data":"ee81946ee3c59c4bf7bac307a6d8b712be32e405bb1352649b713ec38e7415b5"} Nov 02 17:51:24.590384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.590346 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:24.591003 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.590967 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:24.864315 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.864226 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:24.866190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.866163 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:51:24.866395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:24.866369 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:51:25.555072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:25.553146 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:25.555072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:25.553211 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:25.848072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:25.842622 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:51:25.848072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:25.842697 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:51:26.248486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.248394 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:51:26.248752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.248721 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:51:26.554124 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.553517 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:26.554124 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.553574 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:26.700797 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.700204 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:51:26.844070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.840902 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:51:26.844070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:26.840961 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:51:27.546106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:27.543380 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:27.546106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:27.543452 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:27.850071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:27.846414 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:51:27.850071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:27.846478 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:51:28.557115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:28.545483 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:28.557115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:28.545563 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:28.847071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:28.844428 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:51:28.847071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:28.844493 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:51:29.270668 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:51:29.270569 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:51:29.542073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:29.541982 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:51:29.542225 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:29.542063 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:51:29.840275 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:29.840241 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:51:29.840664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:29.840305 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:51:30.546617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.546590 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:30.546896 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.546866 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:30.547488 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.547469 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:51:30.569390 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.569345 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podStartSLOduration=8.56933325 podStartE2EDuration="8.56933325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:24.100858159 +0000 UTC m=+644.199886354" watchObservedRunningTime="2026-11-02 17:51:30.56933325 +0000 UTC m=+650.668361444" Nov 02 17:51:30.844375 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.844341 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:30.844738 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.844671 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:30.845457 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:30.845442 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:51:36.971590 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:36.971551 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv"] Nov 02 17:51:36.982090 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:36.981444 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.001075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:36.997348 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea752f-openshift-default-dockercfg-dsvkg\"" Nov 02 17:51:37.001980 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.001958 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv"] Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016610 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016689 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016720 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016745 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016786 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016861 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016887 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmzw\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016941 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.027952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.016998 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.119019 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.118969 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119338 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119370 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmzw\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119426 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119474 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119563 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119632 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119654 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.119677 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120592 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.120158 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.120730 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.120705 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.121132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.120933 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.121132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.121083 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.121569 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.121545 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.122855 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.122835 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.122951 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.122922 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.135252 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.135163 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmzw\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.147075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.140561 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.330982 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.330944 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:37.706382 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.706348 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv"] Nov 02 17:51:37.707370 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:51:37.707343 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc4e599_6084_4310_a694_8ed1e3d95777.slice/crio-df59f7d56df7c309a821a231b51cca553b926150680c4d8058a11b6f6339a5a0 WatchSource:0}: Error finding container df59f7d56df7c309a821a231b51cca553b926150680c4d8058a11b6f6339a5a0: Status 404 returned error can't find the container with id df59f7d56df7c309a821a231b51cca553b926150680c4d8058a11b6f6339a5a0 Nov 02 17:51:37.709882 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.709847 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:37.709983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.709928 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:37.709983 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:37.709972 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:38.421071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:38.418826 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" event={"ID":"0bc4e599-6084-4310-a694-8ed1e3d95777","Type":"ContainerStarted","Data":"f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5"} Nov 02 17:51:38.421071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:38.418920 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" event={"ID":"0bc4e599-6084-4310-a694-8ed1e3d95777","Type":"ContainerStarted","Data":"df59f7d56df7c309a821a231b51cca553b926150680c4d8058a11b6f6339a5a0"} Nov 02 17:51:39.331497 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:39.331462 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:39.332776 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:39.332749 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:51:39.332942 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:39.332802 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:51:40.331332 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:40.331297 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:51:40.331769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:40.331361 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:51:41.332018 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:41.331987 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:51:41.332405 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:41.332071 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:51:42.331539 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:42.331501 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:51:42.331742 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:42.331552 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:51:43.331914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:43.331876 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:51:43.332368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:43.331941 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:51:44.469768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:44.469113 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:44.469768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:44.469664 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:44.471195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:44.471143 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 17:51:44.498174 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:44.498126 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podStartSLOduration=8.498114008 podStartE2EDuration="8.498114008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:38.457202393 +0000 UTC m=+658.556230601" watchObservedRunningTime="2026-11-02 17:51:44.498114008 +0000 UTC m=+664.597142203" Nov 02 17:51:46.732506 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.732467 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv"] Nov 02 17:51:46.761243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.761216 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv"] Nov 02 17:51:46.761853 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.761835 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.873727 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.873831 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c623cbae-81a6-401d-b9e3-d40200a26ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.873869 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.873894 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.873990 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.874022 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.874077 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874597 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.874120 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.874597 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.874152 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54m4\" (UniqueName: \"kubernetes.io/projected/c623cbae-81a6-401d-b9e3-d40200a26ea7-kube-api-access-j54m4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:46.978745 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:46.978702 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.001207 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.001123 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.001440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.001417 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.001597 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.001580 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.001724 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.001702 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54m4\" (UniqueName: \"kubernetes.io/projected/c623cbae-81a6-401d-b9e3-d40200a26ea7-kube-api-access-j54m4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.001893 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.001877 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.002109 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.002091 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c623cbae-81a6-401d-b9e3-d40200a26ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.002257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.002238 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.002373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.002357 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.003255 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.003232 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.010175 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.003905 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.010328 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.004186 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.010444 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.004554 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.010547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.004914 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c623cbae-81a6-401d-b9e3-d40200a26ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.030099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.028305 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.061719 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.061651 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.063004 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.062979 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c623cbae-81a6-401d-b9e3-d40200a26ea7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.063297 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.063277 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54m4\" (UniqueName: \"kubernetes.io/projected/c623cbae-81a6-401d-b9e3-d40200a26ea7-kube-api-access-j54m4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gg4rv\" (UID: \"c623cbae-81a6-401d-b9e3-d40200a26ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.126067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.125218 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:47.604107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.604077 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv"] Nov 02 17:51:47.606301 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:51:47.606277 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc623cbae_81a6_401d_b9e3_d40200a26ea7.slice/crio-9a57c0e1124d29563f2974ee2d59b8ed0055b9a88ebacef460578b9324c7e97e WatchSource:0}: Error finding container 9a57c0e1124d29563f2974ee2d59b8ed0055b9a88ebacef460578b9324c7e97e: Status 404 returned error can't find the container with id 9a57c0e1124d29563f2974ee2d59b8ed0055b9a88ebacef460578b9324c7e97e Nov 02 17:51:47.608708 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.608675 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:47.608782 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.608742 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:47.608782 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:47.608772 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:48.521989 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:48.521947 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" event={"ID":"c623cbae-81a6-401d-b9e3-d40200a26ea7","Type":"ContainerStarted","Data":"1aff6f6a9d308dd19d8218262cd7bc1fd4ec06a6b1a733e993217e5a01cf54ae"} Nov 02 17:51:48.522397 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:48.522020 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" event={"ID":"c623cbae-81a6-401d-b9e3-d40200a26ea7","Type":"ContainerStarted","Data":"9a57c0e1124d29563f2974ee2d59b8ed0055b9a88ebacef460578b9324c7e97e"} Nov 02 17:51:49.126310 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:49.126273 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:49.127571 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:49.127543 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:49.127706 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:49.127590 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:50.126311 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:50.126273 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:50.126694 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:50.126335 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:51.126362 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:51.126333 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:51.126733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:51.126396 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:52.126778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.126746 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:52.127187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.126809 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:52.175571 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.175525 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podStartSLOduration=6.1755107240000005 podStartE2EDuration="6.175510724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:48.555897648 +0000 UTC m=+668.654925857" watchObservedRunningTime="2026-11-02 17:51:52.175510724 +0000 UTC m=+672.274538918" Nov 02 17:51:52.177294 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.177270 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm"] Nov 02 17:51:52.181005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.180986 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.187742 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.187716 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.187827 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.187784 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.187928 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.187899 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.188027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.188010 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dkw\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.188116 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.188061 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.188116 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.188082 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.188264 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.188186 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.188340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.188289 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.188489 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.188398 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.201704 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.201684 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9dfa27-openshift-default-dockercfg-z57p2\"" Nov 02 17:51:52.214012 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.213990 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm"] Nov 02 17:51:52.289559 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289530 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289611 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dkw\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289656 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289683 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289729 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289813 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289903 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289948 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.289967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289953 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.290203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.289986 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.290203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.290158 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.290314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.290280 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.290354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.290302 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.290509 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.290491 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.291999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.291981 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.292257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.292240 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.307211 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.307192 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dkw\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.309084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.309066 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.432396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.432322 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5"] Nov 02 17:51:52.462343 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.462313 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.465834 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.465755 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5"] Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492762 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmgl\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492811 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492828 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492846 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492891 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492923 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.492956 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.493020 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.493266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.493039 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.495500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.495483 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594487 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594601 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594633 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594756 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmgl\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594804 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594830 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594861 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594927 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.595962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.594984 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.597449 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.596804 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.597449 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.597154 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.597640 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.597509 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.599151 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.599108 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.599521 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.599476 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.601135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.601080 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.602183 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.602141 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.607837 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.607795 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.609895 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.609873 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmgl\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.783715 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.783605 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:52.825908 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.825876 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm"] Nov 02 17:51:52.826586 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:51:52.826543 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735322c1_4325_4305_9b43_4f56770ef13d.slice/crio-29eb00e21a306306b1c8463ef93e332cd827a2121907cbadd9f3b88ee5b2b96c WatchSource:0}: Error finding container 29eb00e21a306306b1c8463ef93e332cd827a2121907cbadd9f3b88ee5b2b96c: Status 404 returned error can't find the container with id 29eb00e21a306306b1c8463ef93e332cd827a2121907cbadd9f3b88ee5b2b96c Nov 02 17:51:52.828659 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.828628 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:52.828750 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.828689 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:52.828750 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:52.828717 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:53.102345 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.102316 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5"] Nov 02 17:51:53.117098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.117033 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:53.117500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.117442 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:53.119965 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.119877 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:51:53.125902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.125879 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:53.125996 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.125926 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:53.577852 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.577802 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" event={"ID":"1e89a018-3e27-4a43-ac81-d429849254a4","Type":"ContainerStarted","Data":"7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe"} Nov 02 17:51:53.578460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.577922 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" event={"ID":"1e89a018-3e27-4a43-ac81-d429849254a4","Type":"ContainerStarted","Data":"afdecb8a655bed6807810d5facf7301728b18f78828525a055efd1ed2cb097d6"} Nov 02 17:51:53.579871 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.579844 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" event={"ID":"735322c1-4325-4305-9b43-4f56770ef13d","Type":"ContainerStarted","Data":"6062cde469c18de70628ee101113a038bd50731848e828649b2c0cb2d10e5aa6"} Nov 02 17:51:53.580001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.579895 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" event={"ID":"735322c1-4325-4305-9b43-4f56770ef13d","Type":"ContainerStarted","Data":"29eb00e21a306306b1c8463ef93e332cd827a2121907cbadd9f3b88ee5b2b96c"} Nov 02 17:51:53.608929 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.608880 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podStartSLOduration=1.6088670120000002 podStartE2EDuration="1.608867012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:53.605852709 +0000 UTC m=+673.704880938" watchObservedRunningTime="2026-11-02 17:51:53.608867012 +0000 UTC m=+673.707895203" Nov 02 17:51:53.648547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.648483 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podStartSLOduration=1.648464663 podStartE2EDuration="1.648464663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:53.645718694 +0000 UTC m=+673.744746914" watchObservedRunningTime="2026-11-02 17:51:53.648464663 +0000 UTC m=+673.747492858" Nov 02 17:51:53.784577 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:53.784539 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:51:54.126484 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.126450 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.126671 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.126504 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:54.497542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.497446 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:51:54.499809 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.499196 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.499809 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.499277 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:51:54.800262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.800171 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.800816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:54.800786 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 17:51:55.126769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:55.126736 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.127011 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:55.126982 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:51:55.498966 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:55.498524 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.498966 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:55.498573 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:51:55.811133 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:55.810413 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.811133 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:55.810470 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 17:51:56.503070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.496446 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:51:56.503070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.496508 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:51:56.788140 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.786464 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:51:56.788140 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.786544 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 17:51:56.808174 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.808148 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:56.809842 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.809401 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:56.810317 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:56.810303 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" Nov 02 17:51:57.496676 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:57.496626 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:51:57.497237 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:57.496694 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:51:57.789126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:57.788457 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:51:57.789126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:57.788523 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 17:51:58.498454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:58.498410 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:51:58.498894 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:58.498469 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:51:58.786066 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:58.785453 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:51:58.786066 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:58.785533 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 17:51:59.283009 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:51:59.282972 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:51:59.497128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:59.497091 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:51:59.497319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:59.497147 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:51:59.784268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:59.784225 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 17:51:59.784768 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:51:59.784295 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 17:52:01.544936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:01.544169 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:52:01.544936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:01.544232 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:01.789808 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:01.789141 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:52:01.789808 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:01.789255 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:02.312881 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:02.312850 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:52:02.314176 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:02.314146 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:52:02.315181 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:02.315156 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:52:02.586128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:02.584363 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:52:02.586128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:02.584989 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:52:02.587084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:02.587057 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:52:06.910791 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.910757 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q"] Nov 02 17:52:06.916333 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.916307 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.941023 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.940997 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q"] Nov 02 17:52:06.967677 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.967650 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.967803 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.967696 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.967847 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.967821 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.967909 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.967893 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.968027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.968004 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.968116 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.968067 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.968175 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.968142 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxnn\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.968274 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.968257 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:06.968322 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:06.968310 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.069881 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.069846 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.069881 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.069894 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070216 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.069935 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070216 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.069982 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070216 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070026 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070216 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070094 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070216 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070122 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxnn\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070255 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070295 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070571 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070499 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070722 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.070845 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.070759 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.071042 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.071012 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.071170 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.071024 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.072318 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.072297 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.072688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.072672 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.085317 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.085289 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.086347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.086326 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxnn\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.230999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.230687 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:07.251319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.251287 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn"] Nov 02 17:52:07.256855 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.256831 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.271987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.271803 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.271987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.271936 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.271987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.271967 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.272231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.272061 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.272231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.272088 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg9z\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.272231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.272117 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.272231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.272154 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.272231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.272196 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.272544 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.272245 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.275526 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.275501 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn"] Nov 02 17:52:07.352115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.352028 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj"] Nov 02 17:52:07.356759 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.356734 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.369383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.369356 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"kube-root-ca.crt\"" Nov 02 17:52:07.369383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.369364 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"openshift-service-ca.crt\"" Nov 02 17:52:07.369721 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.369699 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"default-dockercfg-dtsf8\"" Nov 02 17:52:07.369840 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.369817 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"distinct-w0-f37d16-kserve-self-signed-certs\"" Nov 02 17:52:07.373457 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373433 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj"] Nov 02 17:52:07.373571 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373555 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.373619 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373585 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.373619 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373606 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.373700 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373660 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlpc\" (UniqueName: \"kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.373752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373696 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.373836 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373817 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.373893 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373876 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.373893 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373889 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.373999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373954 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.373999 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.373983 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg9z\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.374122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374010 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.374122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374041 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.374122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374116 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.374270 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374163 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.374270 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374191 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.374363 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374311 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.374624 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.374603 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.375092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.375067 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.375451 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.375428 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.375779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.375761 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.375879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.375833 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.378257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.378232 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.401586 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.401562 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.401877 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.401859 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg9z\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.476219 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476175 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.476419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476250 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlpc\" (UniqueName: \"kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.476419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476287 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.476419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476326 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.476419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476374 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.476419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476415 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.477154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.476701 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.477154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.477089 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.477386 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.477361 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.479150 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.479125 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.479281 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.479259 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.500352 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.499968 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlpc\" (UniqueName: \"kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc\") pod \"distinct-w0-f37d16-kserve-dd98f4658-gj4pj\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.562977 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.562872 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q"] Nov 02 17:52:07.573524 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:07.573273 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5e8853_02ff_4600_b3a4_2fcece5b2d6c.slice/crio-d9e746c2280d46dc7d1e098a6aea2d518e264399021fa0ddcab028db7bf7cbf5 WatchSource:0}: Error finding container d9e746c2280d46dc7d1e098a6aea2d518e264399021fa0ddcab028db7bf7cbf5: Status 404 returned error can't find the container with id d9e746c2280d46dc7d1e098a6aea2d518e264399021fa0ddcab028db7bf7cbf5 Nov 02 17:52:07.580249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.575719 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:07.580249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.575807 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:07.580249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.575853 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:07.580527 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.580506 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:07.674428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.673283 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:07.802134 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.801402 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" event={"ID":"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c","Type":"ContainerStarted","Data":"ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829"} Nov 02 17:52:07.802134 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.801471 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" event={"ID":"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c","Type":"ContainerStarted","Data":"d9e746c2280d46dc7d1e098a6aea2d518e264399021fa0ddcab028db7bf7cbf5"} Nov 02 17:52:07.893107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.893014 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podStartSLOduration=1.8929949069999998 podStartE2EDuration="1.892994907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:07.890783356 +0000 UTC m=+687.989811551" watchObservedRunningTime="2026-11-02 17:52:07.892994907 +0000 UTC m=+687.992023103" Nov 02 17:52:07.899289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.899264 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn"] Nov 02 17:52:07.899780 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:07.899755 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f46651_12fd_4b03_b004_b44d4af7daa5.slice/crio-aa0fbad962a308e608431325b90f53d6d82ae67e5da678232fa8d31099fe4966 WatchSource:0}: Error finding container aa0fbad962a308e608431325b90f53d6d82ae67e5da678232fa8d31099fe4966: Status 404 returned error can't find the container with id aa0fbad962a308e608431325b90f53d6d82ae67e5da678232fa8d31099fe4966 Nov 02 17:52:07.902290 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.902254 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:07.902347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.902336 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:07.902396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:07.902380 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:08.037104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.035623 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj"] Nov 02 17:52:08.231826 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.231793 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:08.233178 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.233145 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 17:52:08.233315 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.233207 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 17:52:08.808775 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.808742 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" event={"ID":"15f46651-12fd-4b03-b004-b44d4af7daa5","Type":"ContainerStarted","Data":"f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c"} Nov 02 17:52:08.809054 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.809021 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" event={"ID":"15f46651-12fd-4b03-b004-b44d4af7daa5","Type":"ContainerStarted","Data":"aa0fbad962a308e608431325b90f53d6d82ae67e5da678232fa8d31099fe4966"} Nov 02 17:52:08.810198 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.810172 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" event={"ID":"5f609015-e6bb-459a-b183-a242d0efcae1","Type":"ContainerStarted","Data":"7fc75bd04d119512a002f2dd3ab7624a5f07bc524735dd5f172878eb09bfb3ce"} Nov 02 17:52:08.810312 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.810210 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" event={"ID":"5f609015-e6bb-459a-b183-a242d0efcae1","Type":"ContainerStarted","Data":"610fada1ea2f17d3200807c6041573337c5e2990d5e9bdb1a458c261f231e0b2"} Nov 02 17:52:08.877071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.876423 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podStartSLOduration=1.876398559 podStartE2EDuration="1.876398559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:08.846358101 +0000 UTC m=+688.945386297" watchObservedRunningTime="2026-11-02 17:52:08.876398559 +0000 UTC m=+688.975426757" Nov 02 17:52:08.884072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:08.882308 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podStartSLOduration=1.882294283 podStartE2EDuration="1.882294283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:08.881421427 +0000 UTC m=+688.980449646" watchObservedRunningTime="2026-11-02 17:52:08.882294283 +0000 UTC m=+688.981322480" Nov 02 17:52:09.231692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:09.231654 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 17:52:09.232131 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:09.231719 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 17:52:09.583901 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:09.583868 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:09.584302 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:09.584277 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:52:09.584452 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:09.584425 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 17:52:09.675164 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:09.675127 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:10.231455 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:10.231419 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 17:52:10.231608 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:10.231475 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 17:52:10.588106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:10.586579 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:52:10.588106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:10.586658 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 17:52:11.232042 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:11.232005 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 17:52:11.232351 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:11.232320 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 17:52:11.584442 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:11.584405 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:52:11.584704 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:11.584678 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 17:52:12.231862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:12.231830 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 17:52:12.232372 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:12.231890 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 17:52:12.591070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:12.584039 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:52:12.591070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:12.584123 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 17:52:13.232023 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:13.231985 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 17:52:13.232441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:13.232077 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 17:52:13.587071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:13.581427 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:52:13.587071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:13.581503 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 17:52:14.459220 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:14.459183 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:14.460517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:14.460497 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:14.461515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:14.461487 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:52:14.581624 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:14.581592 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:52:14.581794 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:14.581676 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 17:52:15.584697 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:15.584669 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:15.585124 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:15.585040 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:15.585692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:15.585671 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 17:52:17.674368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:17.674330 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:17.687233 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:17.687207 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:17.906208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:17.906181 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:52:21.934487 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.934455 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw"] Nov 02 17:52:21.938286 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.938266 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.965106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.965073 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw"] Nov 02 17:52:21.977068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977010 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977112 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77td\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977363 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977461 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977409 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977461 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977438 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977511 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977561 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977552 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977633 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977600 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:21.977633 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:21.977616 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.078758 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078722 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.078758 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078762 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078780 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078810 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078834 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078864 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.078891 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079311 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079224 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079311 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079269 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079422 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079356 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079494 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079475 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079517 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77td\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079848 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079824 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.079962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.079829 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.081417 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.081385 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.081638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.081620 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.087674 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.087650 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.088197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.088180 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77td\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.252321 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.252227 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:22.563955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.563927 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw"] Nov 02 17:52:22.566248 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:22.566216 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3f3ba8_94c1_45d0_a2ea_7fd8421b96d7.slice/crio-81fb8e75d4110af8762114029cc1cd7cce176316a0a0e5bfba4c371fd5958d47 WatchSource:0}: Error finding container 81fb8e75d4110af8762114029cc1cd7cce176316a0a0e5bfba4c371fd5958d47: Status 404 returned error can't find the container with id 81fb8e75d4110af8762114029cc1cd7cce176316a0a0e5bfba4c371fd5958d47 Nov 02 17:52:22.569415 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.569378 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:22.569542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.569447 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:22.569542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:22.569491 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:23.050265 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:23.050225 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" event={"ID":"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7","Type":"ContainerStarted","Data":"159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8"} Nov 02 17:52:23.050714 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:23.050323 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" event={"ID":"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7","Type":"ContainerStarted","Data":"81fb8e75d4110af8762114029cc1cd7cce176316a0a0e5bfba4c371fd5958d47"} Nov 02 17:52:23.256068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:23.255097 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:23.262067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:23.259186 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:52:23.262067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:23.259263 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 17:52:24.253486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:24.253448 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:52:24.254105 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:24.254037 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 17:52:25.259474 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:25.259435 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:52:25.260231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:25.260193 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 17:52:26.254511 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:26.254469 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:52:26.254817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:26.254788 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 17:52:27.255072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:27.253979 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:52:27.255072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:27.254040 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 17:52:28.255025 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:28.254989 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:52:28.255508 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:28.255448 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 17:52:29.253948 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:52:29.253909 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:52:29.570144 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.570061 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:29.571225 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.571106 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:29.572223 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.572197 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:52:29.598170 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.598104 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podStartSLOduration=8.59808704 podStartE2EDuration="8.59808704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:23.086282804 +0000 UTC m=+703.185311000" watchObservedRunningTime="2026-11-02 17:52:29.59808704 +0000 UTC m=+709.697115237" Nov 02 17:52:29.770675 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.770634 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7"] Nov 02 17:52:29.776782 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.776757 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:29.785517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.785489 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"distinct-w6-7b0683-kserve-self-signed-certs\"" Nov 02 17:52:29.787479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.787451 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7"] Nov 02 17:52:29.906815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.906780 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:29.906998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.906824 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:29.906998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.906968 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:29.907131 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.907022 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:29.907192 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.907129 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbw7\" (UniqueName: \"kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:29.907449 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:29.907424 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.011264 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.011212 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.011974 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.011938 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.012197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.012179 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.012337 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.012323 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.012444 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.012430 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbw7\" (UniqueName: \"kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.012750 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.012732 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.013298 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.013278 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.015125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.013605 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.017666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.015504 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.020227 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.020149 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.037417 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.037385 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.051033 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.050999 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbw7\" (UniqueName: \"kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7\") pod \"distinct-w6-7b0683-kserve-6c9f878897-p9mh7\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.095849 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.095817 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:30.400522 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:30.400494 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7"] Nov 02 17:52:30.402627 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:30.402601 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc9f10d9_20b8_404c_bdeb_a64cc32d355f.slice/crio-014d437cd5794116db89037ba65cdfdca42775ae365adc15b3dd8f96e81ed1a4 WatchSource:0}: Error finding container 014d437cd5794116db89037ba65cdfdca42775ae365adc15b3dd8f96e81ed1a4: Status 404 returned error can't find the container with id 014d437cd5794116db89037ba65cdfdca42775ae365adc15b3dd8f96e81ed1a4 Nov 02 17:52:31.117150 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:31.117104 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" event={"ID":"dc9f10d9-20b8-404c-bdeb-a64cc32d355f","Type":"ContainerStarted","Data":"c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942"} Nov 02 17:52:31.117626 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:31.117166 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" event={"ID":"dc9f10d9-20b8-404c-bdeb-a64cc32d355f","Type":"ContainerStarted","Data":"014d437cd5794116db89037ba65cdfdca42775ae365adc15b3dd8f96e81ed1a4"} Nov 02 17:52:31.141350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:31.141281 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podStartSLOduration=2.141260966 podStartE2EDuration="2.141260966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:31.139658422 +0000 UTC m=+711.238686617" watchObservedRunningTime="2026-11-02 17:52:31.141260966 +0000 UTC m=+711.240289165" Nov 02 17:52:32.097110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:32.096140 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:35.214806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.214769 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k"] Nov 02 17:52:35.219143 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.219119 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.236869 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.236838 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"distinct-w3-2a295c-kserve-self-signed-certs\"" Nov 02 17:52:35.238564 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.238536 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k"] Nov 02 17:52:35.262383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.262347 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.262383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.262386 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.262599 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.262573 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.262650 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.262623 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2sg\" (UniqueName: \"kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.262753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.262737 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.262879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.262865 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.363861 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.363821 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.363930 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.363966 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.364084 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.364114 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2sg\" (UniqueName: \"kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.364188 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.364306 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364705 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.364486 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.364871 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.364852 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.368281 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.368255 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.375751 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.375718 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.380806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.380776 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2sg\" (UniqueName: \"kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg\") pod \"distinct-w3-2a295c-kserve-79f5b94658-nr48k\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.531920 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.531842 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:35.829320 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:35.829293 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k"] Nov 02 17:52:35.831549 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:35.831525 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6079b1c1_3ab9_4de0_ad98_d98d27bc60e9.slice/crio-5ea604af6961c543b1381a6b27ab24d37b5261985f4173fec95164ee5948b71d WatchSource:0}: Error finding container 5ea604af6961c543b1381a6b27ab24d37b5261985f4173fec95164ee5948b71d: Status 404 returned error can't find the container with id 5ea604af6961c543b1381a6b27ab24d37b5261985f4173fec95164ee5948b71d Nov 02 17:52:36.148107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:36.148021 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" event={"ID":"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9","Type":"ContainerStarted","Data":"c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24"} Nov 02 17:52:36.148107 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:36.148075 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" event={"ID":"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9","Type":"ContainerStarted","Data":"5ea604af6961c543b1381a6b27ab24d37b5261985f4173fec95164ee5948b71d"} Nov 02 17:52:36.168828 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:36.168783 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podStartSLOduration=1.168770593 podStartE2EDuration="1.168770593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:36.167233852 +0000 UTC m=+716.266262063" watchObservedRunningTime="2026-11-02 17:52:36.168770593 +0000 UTC m=+716.267798789" Nov 02 17:52:36.532167 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:36.532086 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:37.137156 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.137125 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb"] Nov 02 17:52:37.140942 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.140925 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.156552 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.156505 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb"] Nov 02 17:52:37.172755 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.172729 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h"] Nov 02 17:52:37.176937 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.176917 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.187254 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187233 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187365 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187269 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndw4\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187365 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187320 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187365 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187343 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187527 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187407 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187527 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187481 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187621 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187598 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187677 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187655 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.187730 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.187701 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.207482 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.207457 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h"] Nov 02 17:52:37.289036 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.288994 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289036 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289058 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289083 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289111 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289340 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg5g\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289443 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289399 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289481 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289450 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndw4\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289450 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289502 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289600 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289579 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289635 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289617 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289681 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289660 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289735 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289718 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289780 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289794 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.289868 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289855 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289903 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289885 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.289936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.289884 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.290101 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.290080 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.290101 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.290089 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.290256 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.290148 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.290256 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.290173 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.290256 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.290202 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.292042 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.292019 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.292146 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.292032 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.298233 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.298214 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.298611 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.298589 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndw4\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.390957 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.390900 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.390963 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg5g\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.390997 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391023 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391263 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391129 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391263 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391153 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391382 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391293 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391382 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391360 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391485 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391350 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391485 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391422 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391485 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391440 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391485 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391450 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391696 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391517 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.391972 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.391951 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.393389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.393372 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.393554 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.393541 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.400517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.400490 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg5g\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.402305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.402284 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.457572 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.457546 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:37.494330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.494303 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:37.758950 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.758910 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn"] Nov 02 17:52:37.761451 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.761009 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:37.761451 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.761102 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:37.761451 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.761150 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:37.766396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.766374 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb"] Nov 02 17:52:37.766502 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.766484 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.787429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.786077 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn"] Nov 02 17:52:37.803691 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.803660 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.803816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.803721 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.803816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.803747 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.803961 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.803813 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjnz\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.803961 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.803844 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.804168 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.804125 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.815195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.804360 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.815195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.804568 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.815195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.804632 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.826899 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.826142 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h"] Nov 02 17:52:37.833189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.833148 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:37.833297 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.833224 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:37.833297 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.833269 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:37.906202 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906165 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906320 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906227 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906320 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906314 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906405 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906435 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906543 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906515 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjnz\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906596 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906549 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906645 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906610 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.906725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.906696 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.907350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.907235 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.908887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.908209 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.908887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.908284 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.908887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.908609 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.908887 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.908848 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.911262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.911218 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.912602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.912579 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.925389 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.925364 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:37.926759 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:37.926703 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjnz\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:38.093313 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.093222 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:38.174069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.169565 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" event={"ID":"ca212799-029c-4ded-a676-ed706a80533c","Type":"ContainerStarted","Data":"4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41"} Nov 02 17:52:38.174069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.169627 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" event={"ID":"ca212799-029c-4ded-a676-ed706a80533c","Type":"ContainerStarted","Data":"ab9b2a89d6c4f2b556f22f762058eb931e631220b069f90c75cdbd9b03666b04"} Nov 02 17:52:38.174069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.173931 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" event={"ID":"0bb0ff1b-6414-4938-a094-3d0626b7c845","Type":"ContainerStarted","Data":"45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785"} Nov 02 17:52:38.174069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.173978 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" event={"ID":"0bb0ff1b-6414-4938-a094-3d0626b7c845","Type":"ContainerStarted","Data":"ec781799fcd7b0f7ea926dc92336e22ba876cf1bbc1ce5180b17e1a3bc483211"} Nov 02 17:52:38.201068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.198769 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podStartSLOduration=1.1987496069999999 podStartE2EDuration="1.198749607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:38.19662868 +0000 UTC m=+718.295656878" watchObservedRunningTime="2026-11-02 17:52:38.198749607 +0000 UTC m=+718.297777803" Nov 02 17:52:38.234281 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.234236 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podStartSLOduration=1.234222384 podStartE2EDuration="1.234222384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:38.231678775 +0000 UTC m=+718.330706987" watchObservedRunningTime="2026-11-02 17:52:38.234222384 +0000 UTC m=+718.333250579" Nov 02 17:52:38.480749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.480707 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn"] Nov 02 17:52:38.481190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.481165 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:38.481354 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:38.481252 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6916fd93_159d_47f9_bdfa_d87459f8d514.slice/crio-cbd1ca881da3f235b96a67f4ea16d3f1fbc22b1a4d6952f3d1d6bd95b835b3b8 WatchSource:0}: Error finding container cbd1ca881da3f235b96a67f4ea16d3f1fbc22b1a4d6952f3d1d6bd95b835b3b8: Status 404 returned error can't find the container with id cbd1ca881da3f235b96a67f4ea16d3f1fbc22b1a4d6952f3d1d6bd95b835b3b8 Nov 02 17:52:38.487104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.482634 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:38.487104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.482679 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:38.487104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.484397 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:38.487104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.484497 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:38.487104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.484555 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:38.494949 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.494886 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:38.499417 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.499181 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:38.499417 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:38.499233 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:39.200355 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:39.200317 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" event={"ID":"6916fd93-159d-47f9-bdfa-d87459f8d514","Type":"ContainerStarted","Data":"f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8"} Nov 02 17:52:39.200917 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:39.200895 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" event={"ID":"6916fd93-159d-47f9-bdfa-d87459f8d514","Type":"ContainerStarted","Data":"cbd1ca881da3f235b96a67f4ea16d3f1fbc22b1a4d6952f3d1d6bd95b835b3b8"} Nov 02 17:52:39.458939 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:39.458443 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:39.458939 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:39.458506 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:39.497102 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:39.495433 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:39.497102 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:39.495497 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:40.107069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.102924 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:52:40.107069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.102985 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 17:52:40.107069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.104590 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:40.107069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.104642 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:40.296071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.287680 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:40.424763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.424636 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podStartSLOduration=3.424613656 podStartE2EDuration="3.424613656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:39.24737762 +0000 UTC m=+719.346405820" watchObservedRunningTime="2026-11-02 17:52:40.424613656 +0000 UTC m=+720.523641852" Nov 02 17:52:40.498071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.497318 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:40.498071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.497383 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:40.498071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.497795 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:40.498071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.497832 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:40.526848 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:40.526820 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:52:41.097070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:41.096929 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:52:41.097070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:41.096990 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 17:52:41.466223 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:41.466141 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:41.466772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:41.466740 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:41.496930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:41.495549 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:41.496930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:41.495621 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:42.098687 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:42.098651 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:52:42.105184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:42.105126 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 17:52:42.463129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:42.462427 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:42.463129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:42.462503 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:42.513072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:42.497980 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:42.513072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:42.498036 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:43.103072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:43.102478 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:52:43.103072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:43.102538 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 17:52:43.461132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:43.460496 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:43.461132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:43.460561 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:43.495071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:43.494921 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:43.495071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:43.494987 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:44.094517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.094476 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:52:44.095122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.095086 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 17:52:44.460123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.459577 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:52:44.460123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.459653 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:52:44.494937 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.494899 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 17:52:44.495128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.494969 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 17:52:44.925686 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.925647 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps"] Nov 02 17:52:44.932631 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.932608 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:44.946509 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.946475 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps"] Nov 02 17:52:44.947543 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:44.947232 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"distinct-2-w0-f37d16-kserve-self-signed-certs\"" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.093265 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.093843 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.093919 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.094533 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdn9c\" (UniqueName: \"kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.094650 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.094951 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.095006 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.097293 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.095146 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.199864 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.199760 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.200063 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.199862 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn9c\" (UniqueName: \"kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.200063 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.200023 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.201475 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.201355 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.201628 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.201491 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.201628 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.201621 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.201879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.201855 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.202466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.202414 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.202861 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.202830 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.214570 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.214509 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.215860 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.215820 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.216709 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.216669 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn9c\" (UniqueName: \"kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c\") pod \"distinct-2-w0-f37d16-kserve-f4fb84785-9zzps\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.257666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.257327 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:45.601510 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.600109 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:45.632162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.624661 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:45.632684 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.632612 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:45.644079 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.640786 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 17:52:45.660624 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.660553 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:45.720019 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.714209 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps"] Nov 02 17:52:45.808814 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.808788 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:45.809565 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.809496 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:45.813540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:45.813523 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 17:52:46.107080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.107015 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:46.107514 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.107365 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:46.109138 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.109109 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:52:46.324365 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.324327 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" event={"ID":"a8896a5d-a593-410b-9881-325f4bd51b59","Type":"ContainerStarted","Data":"b8d4716326981824e11c8e7a34081393c3b43a66bd15e4a40c23eb70ab86fe70"} Nov 02 17:52:46.324546 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.324373 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" event={"ID":"a8896a5d-a593-410b-9881-325f4bd51b59","Type":"ContainerStarted","Data":"4918800f0c8512bccac3e2f144c023d7be3c707811c448704368ad7cc7ec9adf"} Nov 02 17:52:46.337410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.337384 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:52:46.352484 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:46.352349 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podStartSLOduration=2.352328167 podStartE2EDuration="2.352328167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:46.350997148 +0000 UTC m=+726.450025344" watchObservedRunningTime="2026-11-02 17:52:46.352328167 +0000 UTC m=+726.451356362" Nov 02 17:52:47.258309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:47.258278 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:50.309251 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.300153 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.335081 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.332151 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:52:50.541508 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.541461 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.553076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.551357 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.577172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.575522 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.658218 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.658177 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.710179 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.710123 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:52:50.913289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:50.913180 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:52.386345 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.386308 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2"] Nov 02 17:52:52.390305 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.390284 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.435066 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.432756 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2"] Nov 02 17:52:52.527007 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.526969 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527232 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527026 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527232 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527178 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527232 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfnh\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527494 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527475 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527501 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527527 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527708 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527689 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.527807 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.527782 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.628844 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.628809 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.628888 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629567 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629200 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629578 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfnh\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629590 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl"] Nov 02 17:52:52.629811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629734 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629761 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629929 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629874 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.629982 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.629966 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.630104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.630078 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.631073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.631020 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.631862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.631842 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.631970 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.631950 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.632243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.632219 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.632347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.632252 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.634317 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.634273 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.639839 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.634788 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.639839 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.638507 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.651780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.650569 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.651780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.650737 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfnh\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.661813 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.661780 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl"] Nov 02 17:52:52.731316 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731269 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731493 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731330 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bcf\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731493 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731418 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731493 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731447 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731493 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731464 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731709 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731536 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731709 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731646 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731784 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731748 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.731826 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.731816 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.735731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.735708 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:52.840230 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840202 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.840558 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840539 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.840677 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840663 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bcf\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.840789 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840776 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.840876 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840863 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.840966 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840954 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.841093 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.841078 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.841254 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.841242 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.841388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.841376 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.841513 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.840894 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.841727 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.841270 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.841891 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.841869 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.842003 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.841985 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.842183 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.842165 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.843207 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.843158 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.846211 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.846184 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.850324 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.850300 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bcf\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.850664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.850642 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:52.990082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:52.989956 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:53.036777 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.036746 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2"] Nov 02 17:52:53.038528 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:53.038498 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6b986a_922e_440d_948c_e82d40bdb696.slice/crio-811483d682747b2a84851759455e154c41740b1ee5ecb63d8a2d05da5b76e8ab WatchSource:0}: Error finding container 811483d682747b2a84851759455e154c41740b1ee5ecb63d8a2d05da5b76e8ab: Status 404 returned error can't find the container with id 811483d682747b2a84851759455e154c41740b1ee5ecb63d8a2d05da5b76e8ab Nov 02 17:52:53.041271 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.041239 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:53.041350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.041311 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:53.041388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.041351 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:53.289825 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.289794 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl"] Nov 02 17:52:53.291261 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:53.291238 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa3af74_dd62_4b7c_a04c_64d32b90bc71.slice/crio-67fec5cc39f57e0989d9ec45529c6c2bd8ad25c9f716a1999e9684c9a96448a6 WatchSource:0}: Error finding container 67fec5cc39f57e0989d9ec45529c6c2bd8ad25c9f716a1999e9684c9a96448a6: Status 404 returned error can't find the container with id 67fec5cc39f57e0989d9ec45529c6c2bd8ad25c9f716a1999e9684c9a96448a6 Nov 02 17:52:53.294436 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.294370 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:53.294573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.294490 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:53.294573 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.294541 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:52:53.615019 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.614982 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" event={"ID":"0b6b986a-922e-440d-948c-e82d40bdb696","Type":"ContainerStarted","Data":"b80174bf8db1a15397df4499c0d46d48c16923dcba250eae090c5843c22516f5"} Nov 02 17:52:53.615625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.615599 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" event={"ID":"0b6b986a-922e-440d-948c-e82d40bdb696","Type":"ContainerStarted","Data":"811483d682747b2a84851759455e154c41740b1ee5ecb63d8a2d05da5b76e8ab"} Nov 02 17:52:53.616847 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.616821 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" event={"ID":"3fa3af74-dd62-4b7c-a04c-64d32b90bc71","Type":"ContainerStarted","Data":"7d00bd4c097ffeb53a1cf5f6286571237ce66ba0315b7ad79621c7b2390ce520"} Nov 02 17:52:53.616955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.616872 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" event={"ID":"3fa3af74-dd62-4b7c-a04c-64d32b90bc71","Type":"ContainerStarted","Data":"67fec5cc39f57e0989d9ec45529c6c2bd8ad25c9f716a1999e9684c9a96448a6"} Nov 02 17:52:53.712879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.712823 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podStartSLOduration=1.712810395 podStartE2EDuration="1.712810395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:53.711982699 +0000 UTC m=+733.811010889" watchObservedRunningTime="2026-11-02 17:52:53.712810395 +0000 UTC m=+733.811838590" Nov 02 17:52:53.762068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.762018 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:52:53.818749 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.818674 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podStartSLOduration=1.818656182 podStartE2EDuration="1.818656182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:53.810003181 +0000 UTC m=+733.909031379" watchObservedRunningTime="2026-11-02 17:52:53.818656182 +0000 UTC m=+733.917684377" Nov 02 17:52:53.990670 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:53.990578 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:52:54.742570 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:54.742532 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:52:54.743146 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:54.743115 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:52:54.995128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:54.992943 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:52:54.995128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:54.993007 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 17:52:55.271302 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.265371 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:55.309575 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.309536 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:55.329192 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.329153 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh"] Nov 02 17:52:55.333490 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.333467 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.366071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.365279 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9dfa27\"/\"distinct-2-w1-9dfa27-kserve-self-signed-certs\"" Nov 02 17:52:55.424419 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.424141 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh"] Nov 02 17:52:55.553477 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.553385 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtsn\" (UniqueName: \"kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.562212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.562181 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.562572 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.562548 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.562992 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.562972 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.567166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.567140 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.567466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.567447 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.672852 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.672902 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.672933 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.672983 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtsn\" (UniqueName: \"kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.673038 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.673125 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.673839 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.674365 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.676070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.675270 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.702071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.701657 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.706071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.702834 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.706071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.705994 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtsn\" (UniqueName: \"kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn\") pod \"distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.727082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.726552 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:55.740074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.736383 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:52:55.740074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.736457 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:52:55.894071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.875394 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:52:56.000070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:55.999225 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.005097 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.000577 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 17:52:56.200027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.199984 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh"] Nov 02 17:52:56.200741 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:52:56.200720 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5385c3d7_b069_49ce_9ffb_733703b7e3e3.slice/crio-951fc8079785966e1f889eee2e66dfe2bc210fd765ea91a620d6e7d9d5cf8dd1 WatchSource:0}: Error finding container 951fc8079785966e1f889eee2e66dfe2bc210fd765ea91a620d6e7d9d5cf8dd1: Status 404 returned error can't find the container with id 951fc8079785966e1f889eee2e66dfe2bc210fd765ea91a620d6e7d9d5cf8dd1 Nov 02 17:52:56.694187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.694145 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" event={"ID":"5385c3d7-b069-49ce-9ffb-733703b7e3e3","Type":"ContainerStarted","Data":"b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c"} Nov 02 17:52:56.694377 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.694195 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" event={"ID":"5385c3d7-b069-49ce-9ffb-733703b7e3e3","Type":"ContainerStarted","Data":"951fc8079785966e1f889eee2e66dfe2bc210fd765ea91a620d6e7d9d5cf8dd1"} Nov 02 17:52:56.719540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.719488 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podStartSLOduration=1.719474785 podStartE2EDuration="1.719474785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:56.718812311 +0000 UTC m=+736.817840507" watchObservedRunningTime="2026-11-02 17:52:56.719474785 +0000 UTC m=+736.818503021" Nov 02 17:52:56.729203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.729170 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:52:56.759073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.750155 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.759073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.750222 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:52:56.992104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.991564 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.992104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:56.991625 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 17:52:57.759070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:57.741348 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:52:57.759070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:57.741405 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:52:57.991225 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:57.990420 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:52:57.991225 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:57.990556 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 17:52:58.748070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:58.746269 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:52:58.748070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:58.746341 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:52:58.990785 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:58.990749 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:52:58.991027 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:58.990999 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 17:52:59.285293 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:52:59.285215 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:52:59.736358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:59.736326 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.736538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:59.736378 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:52:59.990883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:59.990808 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.990883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:52:59.990869 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 17:53:00.742136 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:00.742104 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:53:00.742527 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:00.742492 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:53:00.743374 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:00.743352 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 17:53:00.996161 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:00.996090 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:53:00.996602 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:00.996393 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:53:00.997197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:00.997173 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:53:05.727390 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:05.727359 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:53:05.740372 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:05.740345 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:53:05.770945 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:05.770917 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:53:07.918374 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.918335 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k"] Nov 02 17:53:07.925784 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.925757 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k"] Nov 02 17:53:07.926202 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.926156 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.930731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.930678 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.930963 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.930917 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.931241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.931222 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.931365 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.931339 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.931519 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.931500 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.931697 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.931667 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.931968 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.931927 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.932117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.932101 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwcl\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:07.932274 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:07.932256 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.009070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.004192 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp"] Nov 02 17:53:08.033068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.009357 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.036998 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037083 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037116 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037142 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037189 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037244 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037397 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037424 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwcl\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.044072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.037455 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.064959 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.064927 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.092159 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.065494 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.092367 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.065730 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.092463 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.086877 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.092536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.087192 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143225 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143275 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143307 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143379 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143437 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143470 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143499 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143580 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.147076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.143670 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.239705 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.239670 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.245448 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.245413 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.245775 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.245753 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.245930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.245913 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.246089 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.246071 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.246322 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.246300 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.246515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.246495 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.246655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.246639 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.246807 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.246791 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.247059 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.247028 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.247736 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.247713 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.248184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.248164 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.248572 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.248550 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.249373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.249349 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.261106 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.257633 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.288959 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.281009 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.406077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.371303 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.419967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.410394 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.473156 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.451792 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.473156 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.461501 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.473156 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.463700 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.547072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.544935 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwcl\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.592069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.591518 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp"] Nov 02 17:53:08.598289 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.598252 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:08.756733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.756646 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:08.790345 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.790307 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:08.800180 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.800123 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 02 17:53:08.800416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.800392 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:53:08.809200 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.809175 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:08.819127 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.809371 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:53:08.840625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.809112 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:53:08.940287 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:08.940241 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.569123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.551115 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.782429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.782375 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:09.782880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.782854 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:09.783242 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.783121 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:09.784548 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.784521 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp"] Nov 02 17:53:09.798121 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.797063 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.798121 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.797551 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:53:09.800771 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.800666 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.801375 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.801222 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:53:09.821990 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.820227 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:53:09.821990 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.820636 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:53:09.913384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.912923 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:53:09.920395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.914724 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:53:09.939545 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.939515 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k"] Nov 02 17:53:09.989208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:09.987582 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:53:10.025416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.022812 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:10.025416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.022909 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:10.025416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.022963 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:10.025416 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.023225 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:53:10.071139 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.071097 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" event={"ID":"9fc9c67d-05c1-433f-adc3-be3569128b55","Type":"ContainerStarted","Data":"b36182eb76c5268fe806a028593147d17c905f665e6382370f455cb1bf7c8ca0"} Nov 02 17:53:10.071537 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.071516 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:53:10.071752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:10.071737 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:53:11.103190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.102863 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" event={"ID":"d20d8c67-8423-4a9c-be54-62a5fb389495","Type":"ContainerStarted","Data":"57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9"} Nov 02 17:53:11.103190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.102915 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" event={"ID":"d20d8c67-8423-4a9c-be54-62a5fb389495","Type":"ContainerStarted","Data":"5dbaec83faeeb66e9795546d6311422325f906dc184284d3eeb66aaf562fff0c"} Nov 02 17:53:11.104969 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.104942 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" event={"ID":"9fc9c67d-05c1-433f-adc3-be3569128b55","Type":"ContainerStarted","Data":"523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec"} Nov 02 17:53:11.132402 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.132355 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podStartSLOduration=4.13234241 podStartE2EDuration="4.13234241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:11.130919612 +0000 UTC m=+751.229947806" watchObservedRunningTime="2026-11-02 17:53:11.13234241 +0000 UTC m=+751.231370605" Nov 02 17:53:11.176072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.169230 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podStartSLOduration=4.169199465 podStartE2EDuration="4.169199465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:11.168625387 +0000 UTC m=+751.267653581" watchObservedRunningTime="2026-11-02 17:53:11.169199465 +0000 UTC m=+751.268227663" Nov 02 17:53:11.602166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.602117 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:11.603208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.602783 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:53:11.603328 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.603240 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:53:11.765071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.764403 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:11.772187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.772154 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:53:11.772456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:11.772423 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:53:12.601686 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:12.601652 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:53:12.602281 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:12.601706 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:53:12.760456 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:12.760419 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:53:12.760772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:12.760724 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:53:13.608072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:13.602459 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:53:13.608072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:13.602518 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:53:13.762426 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:13.762390 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:53:13.762724 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:13.762695 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:53:14.602308 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:14.602269 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:53:14.602500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:14.602330 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:53:14.764505 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:14.764469 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:53:14.770498 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:14.770093 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:53:15.603313 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.603273 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:53:15.603702 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.603654 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:53:15.792071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.789179 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:53:15.792071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.789259 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:53:15.883194 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.882698 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg"] Nov 02 17:53:15.917420 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.917389 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg"] Nov 02 17:53:15.918896 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.918242 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:15.932032 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:15.931995 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2a295c\"/\"distinct-2-w3-2a295c-kserve-self-signed-certs\"" Nov 02 17:53:16.135125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.131821 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.135125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.131868 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.135125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.132178 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jmk\" (UniqueName: \"kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.135125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.132212 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.135125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.132312 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.135125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.132406 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.289384 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jmk\" (UniqueName: \"kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.289449 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.289502 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.289602 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.289733 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.289765 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.295728 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.295991 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.322073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.296248 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.359076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.335794 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jmk\" (UniqueName: \"kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.360213 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.349702 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.462722 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.462640 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs\") pod \"distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.553080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.553029 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:16.602091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.602037 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:53:16.602319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.602298 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:53:16.759828 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.759743 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:53:16.759828 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.759809 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:53:16.855605 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:53:16.855565 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddb8e678_a415_4baa_afc5_bb57cb3dc07d.slice/crio-9daa841b8e56bc248be12ff03947ffebaa9a3103c41772e120188e5ba09ef0c1 WatchSource:0}: Error finding container 9daa841b8e56bc248be12ff03947ffebaa9a3103c41772e120188e5ba09ef0c1: Status 404 returned error can't find the container with id 9daa841b8e56bc248be12ff03947ffebaa9a3103c41772e120188e5ba09ef0c1 Nov 02 17:53:16.858761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:16.858734 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg"] Nov 02 17:53:17.242593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.242557 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" event={"ID":"ddb8e678-a415-4baa-afc5-bb57cb3dc07d","Type":"ContainerStarted","Data":"01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8"} Nov 02 17:53:17.242806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.242606 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" event={"ID":"ddb8e678-a415-4baa-afc5-bb57cb3dc07d","Type":"ContainerStarted","Data":"9daa841b8e56bc248be12ff03947ffebaa9a3103c41772e120188e5ba09ef0c1"} Nov 02 17:53:17.269192 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.269134 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podStartSLOduration=2.269114999 podStartE2EDuration="2.269114999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:17.264763825 +0000 UTC m=+757.363792021" watchObservedRunningTime="2026-11-02 17:53:17.269114999 +0000 UTC m=+757.368143195" Nov 02 17:53:17.556123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.555499 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:17.844585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.843093 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:17.844585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.844537 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:17.845578 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.845552 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:53:17.935435 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.935365 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:17.935967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.935944 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:17.941067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:17.941029 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:53:22.343504 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.343473 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x"] Nov 02 17:53:22.348309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.348288 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.365616 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.365583 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x"] Nov 02 17:53:22.466697 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.466666 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.466880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.466707 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.466880 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.466826 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.466953 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.466897 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rd7\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.466953 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.466931 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.467018 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.466971 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.467089 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.467071 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.467134 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.467123 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.467217 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.467195 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.568641 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.568605 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.568808 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.568728 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rd7\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.568808 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.568779 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.568883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.568831 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.568935 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.568917 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569024 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.568998 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569186 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569039 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569186 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569098 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569186 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569133 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569375 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569304 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569438 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569369 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569484 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569460 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569605 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569581 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.569719 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.569652 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.571193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.571175 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.571325 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.571309 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.582804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.582783 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.583011 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.582991 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rd7\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.666340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.666274 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:22.712966 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.712933 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg"] Nov 02 17:53:22.718361 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.718335 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.738203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.738180 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg"] Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.771984 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772039 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772080 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772135 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772179 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772238 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772261 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9fz\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772349 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.772924 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.772455 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.873725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.873690 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.873902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.873826 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.873902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.873869 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.873902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.873896 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.873953 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874153 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874091 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874211 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874170 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874211 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874200 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9fz\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874342 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874304 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874341 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874383 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874629 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874603 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874746 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874640 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.874840 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.874822 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.876578 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.876551 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.876779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.876762 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.883482 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.883453 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9fz\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.883482 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.883477 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:22.964514 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.964490 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x"] Nov 02 17:53:22.965924 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:53:22.965901 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48ca71_e254_46fb_b6e5_f5f350caf7ab.slice/crio-4d20176cdbc90cfc202ea97e69a29173485f0cfc66910e915eed836056be7ae0 WatchSource:0}: Error finding container 4d20176cdbc90cfc202ea97e69a29173485f0cfc66910e915eed836056be7ae0: Status 404 returned error can't find the container with id 4d20176cdbc90cfc202ea97e69a29173485f0cfc66910e915eed836056be7ae0 Nov 02 17:53:22.968632 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.968604 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:22.968717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.968667 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:22.968717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:22.968696 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:23.047088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.047041 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:23.285086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.284985 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" event={"ID":"7c48ca71-e254-46fb-b6e5-f5f350caf7ab","Type":"ContainerStarted","Data":"f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4"} Nov 02 17:53:23.285086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.285068 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" event={"ID":"7c48ca71-e254-46fb-b6e5-f5f350caf7ab","Type":"ContainerStarted","Data":"4d20176cdbc90cfc202ea97e69a29173485f0cfc66910e915eed836056be7ae0"} Nov 02 17:53:23.314481 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.314419 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podStartSLOduration=1.314400944 podStartE2EDuration="1.314400944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:23.310012627 +0000 UTC m=+763.409040822" watchObservedRunningTime="2026-11-02 17:53:23.314400944 +0000 UTC m=+763.413429140" Nov 02 17:53:23.349238 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.349195 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg"] Nov 02 17:53:23.382914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.382027 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:23.382914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.382208 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:23.382914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.382254 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:23.667294 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:23.667172 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:24.302217 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:24.302180 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" event={"ID":"c325a4f3-27a3-49b7-a3da-911e936fc65f","Type":"ContainerStarted","Data":"b0e172753644f56f879557c14928772c8535f360b08f57ad5ac823a95d21260a"} Nov 02 17:53:24.302431 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:24.302239 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" event={"ID":"c325a4f3-27a3-49b7-a3da-911e936fc65f","Type":"ContainerStarted","Data":"f9086200199b98d772aa47b23812875a99cb3be8d01dae07cd057f5bc7aeb06f"} Nov 02 17:53:24.359073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:24.355266 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podStartSLOduration=2.355243122 podStartE2EDuration="2.355243122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:24.335772147 +0000 UTC m=+764.434800339" watchObservedRunningTime="2026-11-02 17:53:24.355243122 +0000 UTC m=+764.454271319" Nov 02 17:53:24.688070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:24.687666 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:53:24.688070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:24.687734 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:53:25.058438 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:25.058356 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:25.061313 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:25.060984 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:53:25.061313 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:25.061038 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:53:25.670879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:25.670431 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:53:25.670879 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:25.670495 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:53:26.051125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:26.050373 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.051125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:26.050430 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:53:26.557911 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:26.557102 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:26.610288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:26.610262 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:26.671073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:26.669641 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.671073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:26.669699 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:53:27.058067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:27.057318 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.058067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:27.057375 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:53:27.405830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:27.405802 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:27.688122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:27.688004 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.688600 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:27.688124 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:53:28.057471 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:28.057382 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:53:28.057759 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:28.057730 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:53:28.670177 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:28.670142 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:53:28.670474 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:28.670443 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:53:29.059115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:29.052376 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:53:29.059115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:29.052436 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:53:29.303140 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:53:29.303099 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:53:29.666984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:29.666942 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:53:29.667198 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:29.666997 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:53:30.048319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:30.048228 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:53:30.048319 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:30.048297 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:53:30.672384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:30.672350 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:30.672830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:30.672654 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:30.673590 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:30.673569 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:53:31.074203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:31.074175 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:31.074494 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:31.074471 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:31.077579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:31.077555 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:53:39.530073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:39.528151 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.550074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:39.548131 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.610071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:39.596145 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.766071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:39.765150 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:53:39.905068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:39.902141 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.217121 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.204809 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:53:40.224196 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.224153 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:53:40.345070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.341691 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.513128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.511983 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:53:40.513744 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.513714 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:53:40.531246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.531200 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.576077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.346112 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:53:40.615071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.355276 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.788135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.753758 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.813811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.779913 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.826197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.826143 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:53:40.827956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.780325 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 17:53:40.938072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:40.928620 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.212171 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.212120 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.245072 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:53:41.242312 6343 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.226s" Nov 02 17:53:41.245072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.242465 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:53:41.245072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.242508 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:53:41.257070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.254475 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:53:41.264076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.262376 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:53:41.298069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.282201 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.298069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.282721 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.298069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.296689 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:53:41.298069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.297027 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:53:41.548228 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.548132 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.556069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.552397 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.564194 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.564167 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:53:41.564430 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.564414 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:53:41.578689 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.578637 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.592173 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.592119 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:53:41.601314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.601264 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.609145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.609105 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:53:41.625097 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.621417 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.629398 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.629372 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:41.773098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.756099 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.805160 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.781935 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:53:41.805160 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.782388 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:53:41.832000 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.831957 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.832545 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.832516 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:53:41.853074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.841238 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.853074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.841320 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:53:41.954769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.954715 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.955769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.955743 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:53:41.963090 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:41.961181 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.307070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.306538 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.308393 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.308362 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:53:42.339077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.317117 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.554201 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.554152 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.656135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.567165 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.656135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.597134 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:53:42.656135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.621646 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:53:42.656135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.640276 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:53:42.802072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.775586 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.811253 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.785224 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:53:42.846172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.846111 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.854078 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.850511 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 17:53:42.972123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.947079 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:53:43.001070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.998119 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.001070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.998485 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:53:43.001070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:42.999450 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.316383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.316286 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.318622 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.316617 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.319371 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.317640 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Nov 02 17:53:43.677383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.677328 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.678877 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.678841 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:53:43.679717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.679687 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:53:43.683722 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.683692 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:53:43.684373 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.684348 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:53:43.766214 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.766170 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.809173 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.808764 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.850100 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.824942 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.857273 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.852205 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.857273 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.853358 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.870249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:43.870190 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.012378 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.012267 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.013114 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.013084 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:53:44.013757 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.013723 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:53:44.238355 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.238303 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.343241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.343191 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.345190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.343755 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.694130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.688155 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.694130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.688600 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:53:44.694130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.688847 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:53:44.694130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.689078 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:53:44.694130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.689300 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.813279 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.812143 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.828059 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.826922 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:53:44.856254 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.856209 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.873173 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:44.872120 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.022260 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.022153 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:53:45.026967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.022512 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:53:45.026967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.022549 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.300473 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.300380 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:53:45.346622 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.346576 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:53:45.360081 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.355076 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.546918 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.546881 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:45.649810 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.649566 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.650010 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.640167 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:45.650010 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.649942 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.650241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.650115 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:45.650241 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.650176 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.677866 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.677136 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:45.677866 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.677202 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.705073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.704581 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.705575 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.705218 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:53:45.705575 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.705497 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.729071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.705760 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:53:45.729071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.706020 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:53:45.814195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.814139 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.834162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.830678 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.877074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.875880 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.877074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:45.876253 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:53:46.054142 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.053672 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:53:46.056145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.055126 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.262655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.262604 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.301476 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.301430 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:53:46.353194 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.353153 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.355527 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.355491 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.506103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.505514 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:53:46.571072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.539184 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.571072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.539242 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.571072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.539529 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.727189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.679144 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.727189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.679211 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.727189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.724121 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.803188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.729676 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.803188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.730422 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:53:46.803188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.732980 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.803188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.763571 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 02 17:53:46.809085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.803743 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.809085 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.803809 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.855098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.828180 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.855098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.828238 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.855098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.828322 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.855977 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.855546 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.905288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.887563 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.929454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.929182 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.929454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.929342 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.929454 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:46.929419 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.063132 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.062557 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.330294 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.330267 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:53:47.346165 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.346142 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:53:47.358150 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.358108 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:53:47.360185 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.359289 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:53:47.406874 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.406845 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:53:47.541624 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.541588 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:53:47.563791 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.563759 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:53:47.564919 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.564896 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:53:47.566693 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.566676 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:53:47.580666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.580599 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:53:47.582366 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.582346 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:53:47.585680 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.585660 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:53:47.597515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:47.597493 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:53:52.392991 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.392960 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh"] Nov 02 17:53:52.397079 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.397034 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.413832 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.413802 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh"] Nov 02 17:53:52.533524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533486 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533539 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533635 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftkm\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533659 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533680 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533720 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533710 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533901 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533885 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.533941 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.533921 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.534124 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.534097 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646225 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646333 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646377 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646451 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftkm\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646485 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646522 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646566 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646752 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.646797 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.647420 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.647854 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.647916 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.647907 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.648655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.648469 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.648655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.648623 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.651185 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.651162 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.652270 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.652248 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.660144 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.657683 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftkm\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.660144 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.657833 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:52.711826 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:52.711791 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:53.003472 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.003433 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh"] Nov 02 17:53:53.010183 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:53:53.010153 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9844d25c_81ac_41b4_9e20_7dafd1a64058.slice/crio-c97dce1ca2f9d6c00e8fc26cc983a76975fb254d21fa63e3ee7056ac40434081 WatchSource:0}: Error finding container c97dce1ca2f9d6c00e8fc26cc983a76975fb254d21fa63e3ee7056ac40434081: Status 404 returned error can't find the container with id c97dce1ca2f9d6c00e8fc26cc983a76975fb254d21fa63e3ee7056ac40434081 Nov 02 17:53:53.013945 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.013905 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:53.014086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.013983 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:53.014086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.014018 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:53:53.871162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.871063 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" event={"ID":"9844d25c-81ac-41b4-9e20-7dafd1a64058","Type":"ContainerStarted","Data":"0a0a0a009bcd5144d48d596e26288a6a6a4f45cc544d3bd0378f7a856e1312d9"} Nov 02 17:53:53.871162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.871125 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" event={"ID":"9844d25c-81ac-41b4-9e20-7dafd1a64058","Type":"ContainerStarted","Data":"c97dce1ca2f9d6c00e8fc26cc983a76975fb254d21fa63e3ee7056ac40434081"} Nov 02 17:53:53.893597 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:53.893551 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podStartSLOduration=1.893534636 podStartE2EDuration="1.893534636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:53.891848876 +0000 UTC m=+793.990877072" watchObservedRunningTime="2026-11-02 17:53:53.893534636 +0000 UTC m=+793.992562831" Nov 02 17:53:54.711917 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:54.711889 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:53:54.713468 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:54.713441 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 17:53:54.713581 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:54.713492 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 17:53:55.712987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:55.712952 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 17:53:55.713434 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:55.713003 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 17:53:56.712524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:56.712491 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 17:53:56.712703 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:56.712560 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 17:53:57.712461 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:57.712423 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 17:53:57.712926 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:57.712490 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 17:53:58.713092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:58.713060 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 17:53:58.713522 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:58.713115 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 17:53:59.250900 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:53:59.250857 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:53:59.712413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:59.712377 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 17:53:59.712583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:53:59.712428 6343 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 17:54:00.717256 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:00.717213 6343 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:54:00.717905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:00.717593 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:54:00.718505 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:00.718487 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:54:11.655145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:11.655096 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:54:11.676062 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:11.675336 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.817071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:11.809802 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.842072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:11.830033 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.856291 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:11.856140 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.875272 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:11.874197 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.038752 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.038653 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.039599 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.039553 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:54:12.039812 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.039789 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:54:12.309304 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.309183 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.319424 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.319385 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.373075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.367198 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.658318 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.658265 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.659579 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.658965 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:54:12.681278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.659012 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.681278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.659065 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:54:12.681278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.664173 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:54:12.681278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.679584 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.681278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.679946 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:54:12.813191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.813134 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.814175 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.814152 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:54:12.986309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.912298 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.986309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.913011 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:54:12.986309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.914574 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:54:12.986309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.923394 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:54:12.986309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.970179 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.986309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:12.970796 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:54:13.098073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.050135 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.098073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.050591 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:54:13.098073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.053158 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.098073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.053493 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:54:13.098073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.068497 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.098073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.068864 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:54:13.250145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.246294 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.351072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.320135 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:54:13.395072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.352018 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:54:13.415069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.373403 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.443242 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.443214 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:54:13.674895 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.674849 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.710071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.676894 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:54:13.711442 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.687194 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.712646 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.689003 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:54:13.745243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.703892 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.783441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.748371 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:54:13.783441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.703931 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.783441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.750137 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Nov 02 17:54:13.783441 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.750260 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:54:13.820089 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.815130 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.843075 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.841185 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.917071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.914434 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.917071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:13.914731 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:54:14.104939 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.104894 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:54:14.190073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.140669 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.191486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.146951 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:54:14.198158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.152112 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:54:14.216409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.216278 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.338708 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.338668 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:54:14.356249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.352686 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:54:14.474015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.472316 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:54:14.594065 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.585884 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:54:14.673587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.673500 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:54:14.694068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.676161 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:54:14.695515 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.695481 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:54:14.720941 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.720888 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.747422 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.746299 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:54:14.760092 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.759203 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:54:14.817467 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:14.817420 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 17:54:15.152078 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.137682 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.265350 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.265301 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:54:15.314488 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.314458 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:54:15.606485 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.606441 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:54:15.615388 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.615347 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.661934 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.661891 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.791072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.719264 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:54:15.791072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.721407 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.791072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.748110 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.791072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.764151 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:15.791072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.764217 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.791072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.770596 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.852194 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:15.851213 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 17:54:16.154072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.148148 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.271070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.270086 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:54:16.292072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.288509 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.614077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.610142 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.632008 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.631643 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:54:16.639338 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.632644 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:54:16.655084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.644285 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.656414 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.656033 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.682383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.682331 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:16.682660 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.682611 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.720314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.711750 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.720314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.720113 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.720584 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.720503 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.729859 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.729140 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:16.729859 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.729196 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.729859 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.729267 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.729859 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.729553 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.757111 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.756618 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 02 17:54:16.757111 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.756711 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:54:16.781084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.765130 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:16.781084 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.765188 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.808083 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.781408 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.808083 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.784236 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:16.808083 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.784281 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.886207 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.881371 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:54:16.986848 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:16.895599 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 17:54:17.108368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.070316 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:54:17.188086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.187958 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.271220 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.271176 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.316741 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.316712 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:54:17.322163 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.322142 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:54:17.362854 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.362415 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:54:17.368148 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.368126 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:54:17.390473 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.390451 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:54:17.398898 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.398853 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:54:17.421015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.419814 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:54:17.436763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:17.436742 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:54:26.417686 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.417609 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx"] Nov 02 17:54:26.422782 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.422761 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.427475 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.427456 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:54:26.427659 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.427640 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:54:26.428979 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.428960 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:54:26.428979 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.428966 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d64z4\"" Nov 02 17:54:26.440446 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.440411 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx"] Nov 02 17:54:26.523619 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523588 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.523761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523629 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwd9k\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.523761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523648 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.523761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523715 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.523895 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523843 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.523895 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523875 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.523986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.523973 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.625853 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.625812 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.625853 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.625866 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.626117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.625918 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.626117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.626069 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.626117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.626099 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwd9k\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.626117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.626116 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.626352 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.626130 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.626665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.626638 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.628354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.628320 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.628465 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.628421 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.628528 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.628501 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.628641 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.628624 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.635276 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.635250 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwd9k\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.636164 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.636143 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s42mx\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:26.769710 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:26.769639 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:27.081672 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:27.081645 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx"] Nov 02 17:54:27.083771 ip-10-0-141-84 kubenswrapper[6343]: W1102 17:54:27.083742 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb647ac63_1ff8_43d5_a24f_ce773264ff27.slice/crio-af86c216e640055d34e0100ca6f81a17856884ef3e2e1fa48cf12da9962603b2 WatchSource:0}: Error finding container af86c216e640055d34e0100ca6f81a17856884ef3e2e1fa48cf12da9962603b2: Status 404 returned error can't find the container with id af86c216e640055d34e0100ca6f81a17856884ef3e2e1fa48cf12da9962603b2 Nov 02 17:54:27.085723 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:27.085701 6343 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:54:28.052560 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:28.052531 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" event={"ID":"b647ac63-1ff8-43d5-a24f-ce773264ff27","Type":"ContainerStarted","Data":"af86c216e640055d34e0100ca6f81a17856884ef3e2e1fa48cf12da9962603b2"} Nov 02 17:54:29.250547 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:54:29.250511 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:54:30.237740 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:30.237697 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:30.237881 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:30.237772 6343 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 17:54:31.086426 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:31.086391 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" event={"ID":"b647ac63-1ff8-43d5-a24f-ce773264ff27","Type":"ContainerStarted","Data":"b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23"} Nov 02 17:54:31.087765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:31.087742 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:31.089632 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:31.089605 6343 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-s42mx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:54:31.089763 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:31.089658 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" podUID="b647ac63-1ff8-43d5-a24f-ce773264ff27" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:54:31.113033 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:31.112984 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" podStartSLOduration=1.9622763060000001 podStartE2EDuration="5.112972406s" totalImagesPullingTime="3.1506961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:31.109890837 +0000 UTC m=+831.208919042" watchObservedRunningTime="2026-11-02 17:54:31.112972406 +0000 UTC m=+831.212000601" Nov 02 17:54:32.093405 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:32.093374 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 17:54:40.621070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.620975 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:54:40.673081 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.672155 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.694077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.692862 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.720072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.711725 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.742612 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.741218 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:54:40.768071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.765805 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.776070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.772703 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.834197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:40.834146 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.108068 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.107473 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.170086 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.157178 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.209613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.209565 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.210264 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.209867 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:54:41.263169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.262138 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 17:54:41.282636 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.279143 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.321165 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.321121 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:54:41.473096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.471679 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Nov 02 17:54:41.625203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.625157 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.626357 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.626326 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:54:41.676910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.676866 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:54:41.678227 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.678206 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:54:41.694302 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.694247 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.695446 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.695422 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:54:41.727135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.720173 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.744129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.728040 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:54:41.752840 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.745301 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.761705 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.761672 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:54:41.792172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.774143 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.792172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.774568 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:54:41.792172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.774897 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:54:41.792172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.774958 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:54:41.841976 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.837203 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.841976 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:41.838182 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:54:42.162363 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.162320 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.225077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.214151 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.300717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.274183 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.300717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.278685 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.300717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.279398 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.324197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.317567 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.324931 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.324496 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:54:42.631067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.626176 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.683124 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.680425 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.695014 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.694969 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.742439 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.736934 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:54:42.778324 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.778277 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.779646 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.778571 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.802329 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.778652 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.920666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:42.920543 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.145406 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.145352 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" podUID="80de8a96-1a74-4e82-877a-0cb95a840207" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.270182 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.248754 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.270743 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.270599 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.324407 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.315028 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.324407 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.320235 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 17:54:43.359073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.334204 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.394190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.361506 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" podUID="80de8a96-1a74-4e82-877a-0cb95a840207" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.421296 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.420157 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:54:43.528493 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.528399 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:54:43.587140 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:54:43.579100 6343 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.52s" Nov 02 17:54:43.618118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.599265 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:54:43.618118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.599342 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:54:43.618118 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.599379 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:54:43.625669 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.625639 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:54:43.625946 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.625931 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:54:43.626104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.626091 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:54:43.631104 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.627183 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:54:43.698269 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.698133 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.743234 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.743187 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.744629 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.744593 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.788636 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.788535 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Nov 02 17:54:43.788905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.788881 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.826072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.804134 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.826072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.817862 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.880399 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.879174 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.971888 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:43.971597 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.252072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.227605 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.264278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.254562 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.322344 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.322290 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:54:44.349903 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.349835 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.366697 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.366647 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.421538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.421491 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:54:44.537040 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.536942 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.658262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.658198 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.709340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.709287 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.744542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.744496 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.753074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.749599 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:54:44.803274 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.803161 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.805465 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.805422 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.828077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.824700 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:54:44.975275 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:44.975203 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.298061 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.298000 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:54:45.352262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.352207 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.380073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.354524 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.395072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.387119 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:54:45.452097 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.437108 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:54:45.538088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.519196 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:45.538088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.519266 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.538088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.519707 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:45.538088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.519748 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.554151 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.552865 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.649087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.645518 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:45.649087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.645602 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.661562 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.661168 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 17:54:45.671686 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.671647 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:45.671981 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.671949 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.713400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.713356 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.753304 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.753256 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.760250 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.760206 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.813135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.812362 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:54:45.817073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.816162 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.835081 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.835017 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:54:45.994232 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:45.994176 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.109190 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.109144 6343 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.109478 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.109448 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.285195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.285143 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 02 17:54:46.304308 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.304255 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:54:46.355201 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.355150 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.389096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.388980 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.428187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.428144 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.452202 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.452155 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.476074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.467692 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.476074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.467753 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.586254 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.576308 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.597546 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.576385 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 02 17:54:46.705494 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.663563 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.757073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.664451 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.757073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.706362 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.757073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.728201 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.757073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.728262 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.759803 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.759707 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.760356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.760313 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.761257 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.761233 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.767424 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.767385 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.774900 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.761969 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:54:46.782064 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.761925 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:54:46.820080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.816186 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.820080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.816256 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.820347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.820129 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:54:46.844073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.824882 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.903071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.843779 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.903268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.896916 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.903268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.903192 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.903268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.896995 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.903268 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:46.903263 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.006814 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.006667 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.165730 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.165688 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.166016 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.165986 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.166838 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.166812 6343 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.166994 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.166973 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.363072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.362277 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.541295 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.447295 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.541295 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.447376 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.541295 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.515177 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:54:47.633128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.544974 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.705203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.580823 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.705203 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.700263 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 17:54:47.706783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.706733 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.766688 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.766632 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.778076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.776134 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:54:47.778910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.778830 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.838096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.822066 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.913020 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.912917 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:54:47.934884 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:47.914387 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:54:48.064540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.055139 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.294217 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.287024 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:54:48.314661 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.313700 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:54:48.375070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.369022 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:54:48.401070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.399904 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:54:48.420110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.418456 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.653987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.547761 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:54:48.653987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.552141 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.653987 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.553440 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.684200 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.684158 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:54:48.701170 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.701126 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.753348 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.701243 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 02 17:54:48.795015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.714677 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:54:48.795329 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.795311 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:54:48.796211 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.781303 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:54:48.797347 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.784128 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.823298 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.822366 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.853576 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.853552 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:54:48.921317 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.920290 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:54:48.982039 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.982014 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:54:48.982417 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:48.982402 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:54:49.033660 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:49.033632 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:54:49.051022 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:49.050994 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:54:49.052705 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:49.052679 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:54:49.056386 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:49.056367 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:54:49.064291 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:54:49.064271 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:54:59.246106 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:54:59.246070 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:55:16.264072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.263133 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.437409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.437347 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.437773 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.437746 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.548249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.548148 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.550407 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.548816 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 02 17:55:16.674072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.674001 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.692192 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.687504 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:16.692192 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.687566 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.699865 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.699817 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:55:16.718177 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.718127 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.735208 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.735159 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.761072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.760972 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.761362 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.761330 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 17:55:16.804173 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.804069 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.806330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.806298 6343 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:16.806562 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.806536 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.851500 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.851451 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:55:16.898184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.898143 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:16.898473 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:16.898446 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.003204 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.003154 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.348087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.345201 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:55:17.352177 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.352137 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.529240 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.492188 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:55:17.529240 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.492941 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:55:17.529240 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.495130 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.529240 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.495427 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:55:17.574731 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.574692 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.578197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.578158 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.700658 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.578554 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:55:17.700658 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.691586 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.700658 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.692028 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:55:17.720965 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.720915 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.744381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.723988 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:55:17.744381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.724989 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.744381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.737416 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:55:17.769727 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.769681 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.770323 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.770291 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.770503 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.770482 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:55:17.770583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.770561 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:55:17.817575 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.810486 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:55:17.862074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.862017 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:55:17.870073 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.867106 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.874891 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.874852 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.889072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.881578 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:55:17.991409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:17.988558 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dxh8z" podUID="80de8a96-1a74-4e82-877a-0cb95a840207" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.027169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.004406 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.027169 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.005767 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:55:18.359800 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.359223 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.359800 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.359689 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:55:18.367811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.361680 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.367811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.363110 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:55:18.443096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.393467 6343 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:18.443096 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.393530 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.473093 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.444982 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Nov 02 17:55:18.503279 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.503236 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.524171 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.524126 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:55:18.617574 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.590061 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.694303 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.692251 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.694303 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.693458 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.721290 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.717705 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:55:18.722984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.722391 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 17:55:18.735001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.734964 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:55:18.763410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:18.763378 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:55:19.050902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.047679 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 17:55:19.050902 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.026399 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.065947 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.065903 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.068128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.068090 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:55:19.069499 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.069231 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 17:55:19.093240 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.071220 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.093240 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.089309 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:55:19.397067 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.379105 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:55:19.478199 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.432155 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.525957 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.525918 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.533182 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.533127 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.605486 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.605162 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.735121 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.726623 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.745149 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:19.736033 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:55:20.055009 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.054908 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.071570 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.071527 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.072133 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.072103 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 17:55:20.073583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.073559 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 17:55:20.127247 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.127191 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.484138 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.484109 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:55:20.521942 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.521896 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.533187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.533155 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 17:55:20.591711 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.591600 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.625069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.624506 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:55:20.688870 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.688826 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.760126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.754133 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:55:20.830076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.821400 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.831384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.831351 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.925276 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:20.925233 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.057243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.057146 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.086281 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.086249 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 17:55:21.104665 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.104617 6343 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.112655 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.112634 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:55:21.129018 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.128989 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.214392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.214355 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.222606 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.222582 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:55:21.223985 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.223829 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:55:21.224829 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.224804 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 17:55:21.232089 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.232071 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:55:21.232747 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.232550 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:55:21.232931 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.232917 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:55:21.263550 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.263514 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:55:21.267010 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:21.266981 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:55:29.202398 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:29.202367 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7"] Nov 02 17:55:29.229624 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:29.229593 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw"] Nov 02 17:55:29.258119 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:55:29.258083 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:55:40.155158 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:40.155128 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:55:40.158890 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:40.158868 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 17:55:44.251071 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.250813 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" containerID="cri-o://c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942" gracePeriod=45 Nov 02 17:55:44.297074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.293338 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" containerID="cri-o://72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94" gracePeriod=45 Nov 02 17:55:44.607910 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.607880 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:55:44.665098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.659041 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:55:44.765551 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765519 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm\") pod \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " Nov 02 17:55:44.765761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765634 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache\") pod \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " Nov 02 17:55:44.765761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765673 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir\") pod \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " Nov 02 17:55:44.765761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765731 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs\") pod \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " Nov 02 17:55:44.765998 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765971 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache" pod "dc9f10d9-20b8-404c-bdeb-a64cc32d355f" (UID: "dc9f10d9-20b8-404c-bdeb-a64cc32d355f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.766088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765828 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir\") pod \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " Nov 02 17:55:44.766088 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.765981 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir" pod "dc9f10d9-20b8-404c-bdeb-a64cc32d355f" (UID: "dc9f10d9-20b8-404c-bdeb-a64cc32d355f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.766197 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766108 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home\") pod \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " Nov 02 17:55:44.766255 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766237 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7\" (UniqueName: \"kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7\") pod \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\" (UID: \"dc9f10d9-20b8-404c-bdeb-a64cc32d355f\") " Nov 02 17:55:44.766255 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766242 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir" pod "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" (UID: "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.766356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766266 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home\") pod \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " Nov 02 17:55:44.766356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766309 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache\") pod \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " Nov 02 17:55:44.766477 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766383 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm\") pod \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " Nov 02 17:55:44.766477 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766448 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs\") pod \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " Nov 02 17:55:44.766583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766573 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v\" (UniqueName: \"kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v\") pod \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\" (UID: \"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5\") " Nov 02 17:55:44.766635 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766585 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home" pod "dc9f10d9-20b8-404c-bdeb-a64cc32d355f" (UID: "dc9f10d9-20b8-404c-bdeb-a64cc32d355f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.766751 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766690 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home" pod "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" (UID: "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.766849 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.766827 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache" pod "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" (UID: "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.767401 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.767374 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.767530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.767406 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.767530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.767425 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.767530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.767439 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.767530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.767452 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.767530 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.767464 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.769538 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.769510 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v" pod "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" (UID: "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5"). InnerVolumeSpecName "kube-api-access-dzz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:44.769790 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.769768 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7" pod "dc9f10d9-20b8-404c-bdeb-a64cc32d355f" (UID: "dc9f10d9-20b8-404c-bdeb-a64cc32d355f"). InnerVolumeSpecName "kube-api-access-7hbw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:44.769889 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.769842 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm" pod "dc9f10d9-20b8-404c-bdeb-a64cc32d355f" (UID: "dc9f10d9-20b8-404c-bdeb-a64cc32d355f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.770450 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.769878 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm" pod "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" (UID: "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.770450 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.769940 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs" pod "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" (UID: "cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:44.770577 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.769960 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs" pod "dc9f10d9-20b8-404c-bdeb-a64cc32d355f" (UID: "dc9f10d9-20b8-404c-bdeb-a64cc32d355f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:44.871135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.871097 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbw7\" (UniqueName: \"kubernetes.io/projected/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-kube-api-access-7hbw7\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.871135 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.871136 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.872077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.871158 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.872077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.871172 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzz5v\" (UniqueName: \"kubernetes.io/projected/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5-kube-api-access-dzz5v\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.872077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.871185 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.872077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.871204 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9f10d9-20b8-404c-bdeb-a64cc32d355f-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.909072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.907845 6343 generic.go:356] "Generic (PLEG): container finished" podID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerID="c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942" exitCode=0 Nov 02 17:55:44.909072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.907961 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" Nov 02 17:55:44.909072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.908095 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" event={"ID":"dc9f10d9-20b8-404c-bdeb-a64cc32d355f","Type":"ContainerDied","Data":"c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942"} Nov 02 17:55:44.909072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.908140 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" event={"ID":"dc9f10d9-20b8-404c-bdeb-a64cc32d355f","Type":"ContainerDied","Data":"014d437cd5794116db89037ba65cdfdca42775ae365adc15b3dd8f96e81ed1a4"} Nov 02 17:55:44.909072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.908164 6343 scope.go:122] "RemoveContainer" containerID="c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942" Nov 02 17:55:44.917819 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.913868 6343 generic.go:356] "Generic (PLEG): container finished" podID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerID="72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94" exitCode=0 Nov 02 17:55:44.917819 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.913917 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" event={"ID":"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5","Type":"ContainerDied","Data":"72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94"} Nov 02 17:55:44.917819 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.913952 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" event={"ID":"cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5","Type":"ContainerDied","Data":"fc4a94ee118acc236d0fe0941f30d2adf172ac023cd95e1d8a05a5c3e32405f3"} Nov 02 17:55:44.917819 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.914019 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" Nov 02 17:55:44.943395 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.943105 6343 scope.go:122] "RemoveContainer" containerID="c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942" Nov 02 17:55:44.943849 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:55:44.943656 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942\": container with ID starting with c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942 not found: ID does not exist" containerID="c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942" Nov 02 17:55:44.943849 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.943689 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942"} err="failed to get container status \"c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942\": rpc error: code = NotFound desc = could not find container \"c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942\": container with ID starting with c73a2c426105cfb5aeba0d8481805a3c2fd3d7cc14afa8a690e8b1e9d5626942 not found: ID does not exist" Nov 02 17:55:44.948120 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.943712 6343 scope.go:122] "RemoveContainer" containerID="72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94" Nov 02 17:55:44.980151 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.980110 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw"] Nov 02 17:55:44.984878 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.984852 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw"] Nov 02 17:55:44.996260 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.996235 6343 scope.go:122] "RemoveContainer" containerID="72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94" Nov 02 17:55:44.998962 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:55:44.998931 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94\": container with ID starting with 72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94 not found: ID does not exist" containerID="72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94" Nov 02 17:55:44.999129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:44.998975 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94"} err="failed to get container status \"72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94\": rpc error: code = NotFound desc = could not find container \"72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94\": container with ID starting with 72876b927ba5da55adcad8e71ad65548c1177cde64f6014916ddea6fd5eb1a94 not found: ID does not exist" Nov 02 17:55:45.006178 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:45.006144 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7"] Nov 02 17:55:45.010950 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:45.010925 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7"] Nov 02 17:55:45.525984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:45.525935 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-w6-7b0683-kserve-6c9f878897-p9mh7" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:45.591798 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:45.591753 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/simulator-w6-7b0683-kserve-85d9954979-rvzpw" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:46.042314 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:46.042273 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" path="/var/lib/kubelet/pods/cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5/volumes" Nov 02 17:55:46.043093 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:55:46.043036 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" path="/var/lib/kubelet/pods/dc9f10d9-20b8-404c-bdeb-a64cc32d355f/volumes" Nov 02 17:55:59.251173 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:55:59.251125 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:56:02.197325 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.197289 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp"] Nov 02 17:56:02.198843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.197514 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" containerID="cri-o://523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec" gracePeriod=30 Nov 02 17:56:02.212476 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.212437 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5"] Nov 02 17:56:02.212853 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.212720 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" containerID="cri-o://51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb" gracePeriod=30 Nov 02 17:56:02.219341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.218830 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn"] Nov 02 17:56:02.219341 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.219136 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" containerID="cri-o://f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8" gracePeriod=30 Nov 02 17:56:02.847074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.846841 6343 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:02.847074 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:02.846897 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:07.534327 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.534286 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:56:07.585886 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.585860 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:56:07.619426 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.619369 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.619631 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.619432 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:56:07.619838 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.619794 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.620187 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620161 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:07.620550 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620526 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.620670 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620565 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.620670 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620627 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.620670 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620661 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.620850 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620690 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.620850 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620796 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.620962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620869 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.620962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620907 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.620962 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620950 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.621246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.620980 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.621246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621014 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.621246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621122 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.621246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621159 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.621246 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621233 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621286 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621316 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy\") pod \"9fc9c67d-05c1-433f-adc3-be3569128b55\" (UID: \"9fc9c67d-05c1-433f-adc3-be3569128b55\") " Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621410 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo\") pod \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\" (UID: \"1e7f7261-02e4-40a3-b507-112ce6ce4da4\") " Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621496 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621894 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.621972 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.622608 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.622632 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.622647 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9c67d-05c1-433f-adc3-be3569128b55-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.622660 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.622675 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.622729 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.623154 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.623368 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.623546 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.623783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.623703 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.629471 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.629426 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:07.630569 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.630531 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.634470 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.634424 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.634592 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.634472 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.634592 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.634506 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.634592 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.634514 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "kube-api-access-j8bqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.634904 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.634880 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo" pod "1e7f7261-02e4-40a3-b507-112ce6ce4da4" (UID: "1e7f7261-02e4-40a3-b507-112ce6ce4da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:07.636990 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.636955 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz" pod "9fc9c67d-05c1-433f-adc3-be3569128b55" (UID: "9fc9c67d-05c1-433f-adc3-be3569128b55"). InnerVolumeSpecName "kube-api-access-bnfpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723290 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723371 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723434 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723514 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723543 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723637 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723684 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723715 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723812 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.723923 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs\") pod \"6916fd93-159d-47f9-bdfa-d87459f8d514\" (UID: \"6916fd93-159d-47f9-bdfa-d87459f8d514\") " Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724006 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724391 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724398 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724946 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724962 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724971 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724980 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724989 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bqm\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-kube-api-access-j8bqm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.724999 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6916fd93-159d-47f9-bdfa-d87459f8d514-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725007 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725015 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725023 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725031 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725039 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7f7261-02e4-40a3-b507-112ce6ce4da4-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725062 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725072 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-kube-api-access-bnfpz\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725082 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7f7261-02e4-40a3-b507-112ce6ce4da4-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725090 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725099 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc9c67d-05c1-433f-adc3-be3569128b55-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.725110 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc9c67d-05c1-433f-adc3-be3569128b55-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.727103 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.726868 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.728906 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.727223 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:07.728906 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.727338 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "kube-api-access-pxjnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.728906 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.727623 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.729146 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.729118 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs" pod "6916fd93-159d-47f9-bdfa-d87459f8d514" (UID: "6916fd93-159d-47f9-bdfa-d87459f8d514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.826170 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.826143 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.826170 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.826171 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.826353 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.826181 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.826353 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.826192 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxjnz\" (UniqueName: \"kubernetes.io/projected/6916fd93-159d-47f9-bdfa-d87459f8d514-kube-api-access-pxjnz\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.826353 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:07.826202 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6916fd93-159d-47f9-bdfa-d87459f8d514-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:08.133157 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.133079 6343 generic.go:356] "Generic (PLEG): container finished" podID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerID="523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec" exitCode=0 Nov 02 17:56:08.133157 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.133128 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" Nov 02 17:56:08.133359 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.133170 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" event={"ID":"9fc9c67d-05c1-433f-adc3-be3569128b55","Type":"ContainerDied","Data":"523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec"} Nov 02 17:56:08.133359 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.133225 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp" event={"ID":"9fc9c67d-05c1-433f-adc3-be3569128b55","Type":"ContainerDied","Data":"b36182eb76c5268fe806a028593147d17c905f665e6382370f455cb1bf7c8ca0"} Nov 02 17:56:08.133359 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.133254 6343 scope.go:122] "RemoveContainer" containerID="523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec" Nov 02 17:56:08.134849 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.134824 6343 generic.go:356] "Generic (PLEG): container finished" podID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerID="51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb" exitCode=0 Nov 02 17:56:08.134975 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.134862 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" Nov 02 17:56:08.134975 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.134910 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" event={"ID":"1e7f7261-02e4-40a3-b507-112ce6ce4da4","Type":"ContainerDied","Data":"51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb"} Nov 02 17:56:08.134975 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.134964 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5" event={"ID":"1e7f7261-02e4-40a3-b507-112ce6ce4da4","Type":"ContainerDied","Data":"ee81946ee3c59c4bf7bac307a6d8b712be32e405bb1352649b713ec38e7415b5"} Nov 02 17:56:08.136756 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.136735 6343 generic.go:356] "Generic (PLEG): container finished" podID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerID="f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8" exitCode=0 Nov 02 17:56:08.136882 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.136758 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" event={"ID":"6916fd93-159d-47f9-bdfa-d87459f8d514","Type":"ContainerDied","Data":"f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8"} Nov 02 17:56:08.136882 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.136804 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" event={"ID":"6916fd93-159d-47f9-bdfa-d87459f8d514","Type":"ContainerDied","Data":"cbd1ca881da3f235b96a67f4ea16d3f1fbc22b1a4d6952f3d1d6bd95b835b3b8"} Nov 02 17:56:08.136882 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.136838 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn" Nov 02 17:56:08.158290 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.158259 6343 scope.go:122] "RemoveContainer" containerID="523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec" Nov 02 17:56:08.158897 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:08.158865 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec\": container with ID starting with 523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec not found: ID does not exist" containerID="523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec" Nov 02 17:56:08.159015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.158908 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec"} err="failed to get container status \"523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec\": rpc error: code = NotFound desc = could not find container \"523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec\": container with ID starting with 523c80e4aada55025cbf96046c9db8a8401855bd97c24092e4f176388fb87eec not found: ID does not exist" Nov 02 17:56:08.159015 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.158932 6343 scope.go:122] "RemoveContainer" containerID="51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb" Nov 02 17:56:08.168072 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.168028 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn"] Nov 02 17:56:08.174639 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.174558 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-gf6pn"] Nov 02 17:56:08.174713 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.174681 6343 scope.go:122] "RemoveContainer" containerID="51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb" Nov 02 17:56:08.175021 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:08.175000 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb\": container with ID starting with 51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb not found: ID does not exist" containerID="51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb" Nov 02 17:56:08.175131 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.175029 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb"} err="failed to get container status \"51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb\": rpc error: code = NotFound desc = could not find container \"51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb\": container with ID starting with 51c989e683b7111ce4e25e6ebca4a5f58018c4832ab473893f5381e75b5612cb not found: ID does not exist" Nov 02 17:56:08.175131 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.175091 6343 scope.go:122] "RemoveContainer" containerID="f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8" Nov 02 17:56:08.189550 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.189522 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp"] Nov 02 17:56:08.189863 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.189743 6343 scope.go:122] "RemoveContainer" containerID="f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8" Nov 02 17:56:08.190673 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:08.190639 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8\": container with ID starting with f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8 not found: ID does not exist" containerID="f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8" Nov 02 17:56:08.190875 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.190707 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8"} err="failed to get container status \"f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8\": rpc error: code = NotFound desc = could not find container \"f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8\": container with ID starting with f5922a1889eb61239d9ed5f6aaaa7b0b8458007aa36c4591408858bd9fd67ec8 not found: ID does not exist" Nov 02 17:56:08.194937 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.194916 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-zlvcp"] Nov 02 17:56:08.208466 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.208438 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5"] Nov 02 17:56:08.214700 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:08.214675 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-hsgk5"] Nov 02 17:56:10.020623 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:10.020590 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" path="/var/lib/kubelet/pods/1e7f7261-02e4-40a3-b507-112ce6ce4da4/volumes" Nov 02 17:56:10.021097 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:10.021080 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" path="/var/lib/kubelet/pods/6916fd93-159d-47f9-bdfa-d87459f8d514/volumes" Nov 02 17:56:10.021519 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:10.021506 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" path="/var/lib/kubelet/pods/9fc9c67d-05c1-433f-adc3-be3569128b55/volumes" Nov 02 17:56:29.247746 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:29.247704 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:56:41.322593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:41.322559 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps"] Nov 02 17:56:41.327537 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:41.327509 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj"] Nov 02 17:56:42.636017 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:42.635979 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf"] Nov 02 17:56:42.649327 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:42.649220 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2"] Nov 02 17:56:48.148108 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:48.143896 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p"] Nov 02 17:56:53.144801 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:53.144752 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerName="payload-processing" containerID="cri-o://67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0" gracePeriod=25 Nov 02 17:56:54.382258 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:54.382222 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:56:56.346033 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.345975 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" containerID="cri-o://b8d4716326981824e11c8e7a34081393c3b43a66bd15e4a40c23eb70ab86fe70" gracePeriod=45 Nov 02 17:56:56.348434 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.348400 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" containerID="cri-o://7fc75bd04d119512a002f2dd3ab7624a5f07bc524735dd5f172878eb09bfb3ce" gracePeriod=45 Nov 02 17:56:56.641193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.641163 6343 generic.go:356] "Generic (PLEG): container finished" podID="a8896a5d-a593-410b-9881-325f4bd51b59" containerID="b8d4716326981824e11c8e7a34081393c3b43a66bd15e4a40c23eb70ab86fe70" exitCode=0 Nov 02 17:56:56.641340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.641240 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" event={"ID":"a8896a5d-a593-410b-9881-325f4bd51b59","Type":"ContainerDied","Data":"b8d4716326981824e11c8e7a34081393c3b43a66bd15e4a40c23eb70ab86fe70"} Nov 02 17:56:56.642798 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.642769 6343 generic.go:356] "Generic (PLEG): container finished" podID="5f609015-e6bb-459a-b183-a242d0efcae1" containerID="7fc75bd04d119512a002f2dd3ab7624a5f07bc524735dd5f172878eb09bfb3ce" exitCode=0 Nov 02 17:56:56.642954 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.642937 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" event={"ID":"5f609015-e6bb-459a-b183-a242d0efcae1","Type":"ContainerDied","Data":"7fc75bd04d119512a002f2dd3ab7624a5f07bc524735dd5f172878eb09bfb3ce"} Nov 02 17:56:56.643005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.642971 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" event={"ID":"5f609015-e6bb-459a-b183-a242d0efcae1","Type":"ContainerDied","Data":"610fada1ea2f17d3200807c6041573337c5e2990d5e9bdb1a458c261f231e0b2"} Nov 02 17:56:56.643005 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.642984 6343 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610fada1ea2f17d3200807c6041573337c5e2990d5e9bdb1a458c261f231e0b2" Nov 02 17:56:56.643129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.643027 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:56:56.678642 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.678618 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:56:56.723111 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723083 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir\") pod \"5f609015-e6bb-459a-b183-a242d0efcae1\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723186 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc\" (UniqueName: \"kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc\") pod \"5f609015-e6bb-459a-b183-a242d0efcae1\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723240 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache\") pod \"5f609015-e6bb-459a-b183-a242d0efcae1\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723279 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm\") pod \"5f609015-e6bb-459a-b183-a242d0efcae1\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723350 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs\") pod \"5f609015-e6bb-459a-b183-a242d0efcae1\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723378 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home\") pod \"5f609015-e6bb-459a-b183-a242d0efcae1\" (UID: \"5f609015-e6bb-459a-b183-a242d0efcae1\") " Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723381 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir" pod "5f609015-e6bb-459a-b183-a242d0efcae1" (UID: "5f609015-e6bb-459a-b183-a242d0efcae1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723615 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home" pod "5f609015-e6bb-459a-b183-a242d0efcae1" (UID: "5f609015-e6bb-459a-b183-a242d0efcae1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.723780 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache" pod "5f609015-e6bb-459a-b183-a242d0efcae1" (UID: "5f609015-e6bb-459a-b183-a242d0efcae1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.724336 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.724359 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.726195 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.724373 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.726804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.726598 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm" pod "5f609015-e6bb-459a-b183-a242d0efcae1" (UID: "5f609015-e6bb-459a-b183-a242d0efcae1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.726804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.726745 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc" pod "5f609015-e6bb-459a-b183-a242d0efcae1" (UID: "5f609015-e6bb-459a-b183-a242d0efcae1"). InnerVolumeSpecName "kube-api-access-xvlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:56.732083 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.728301 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs" pod "5f609015-e6bb-459a-b183-a242d0efcae1" (UID: "5f609015-e6bb-459a-b183-a242d0efcae1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:56.829378 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829342 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir\") pod \"a8896a5d-a593-410b-9881-325f4bd51b59\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " Nov 02 17:56:56.829577 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829433 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs\") pod \"a8896a5d-a593-410b-9881-325f4bd51b59\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " Nov 02 17:56:56.829577 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829542 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm\") pod \"a8896a5d-a593-410b-9881-325f4bd51b59\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " Nov 02 17:56:56.829780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829590 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home\") pod \"a8896a5d-a593-410b-9881-325f4bd51b59\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " Nov 02 17:56:56.829780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829639 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir" pod "a8896a5d-a593-410b-9881-325f4bd51b59" (UID: "a8896a5d-a593-410b-9881-325f4bd51b59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.829780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829715 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c\" (UniqueName: \"kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c\") pod \"a8896a5d-a593-410b-9881-325f4bd51b59\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " Nov 02 17:56:56.829957 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829880 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home" pod "a8896a5d-a593-410b-9881-325f4bd51b59" (UID: "a8896a5d-a593-410b-9881-325f4bd51b59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.829957 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.829920 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache\") pod \"a8896a5d-a593-410b-9881-325f4bd51b59\" (UID: \"a8896a5d-a593-410b-9881-325f4bd51b59\") " Nov 02 17:56:56.830437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830410 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache" pod "a8896a5d-a593-410b-9881-325f4bd51b59" (UID: "a8896a5d-a593-410b-9881-325f4bd51b59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.830716 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830698 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlpc\" (UniqueName: \"kubernetes.io/projected/5f609015-e6bb-459a-b183-a242d0efcae1-kube-api-access-xvlpc\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.830804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830720 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f609015-e6bb-459a-b183-a242d0efcae1-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.830804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830735 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.830804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830749 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.830804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830764 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f609015-e6bb-459a-b183-a242d0efcae1-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.830804 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.830776 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.832666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.832640 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c" pod "a8896a5d-a593-410b-9881-325f4bd51b59" (UID: "a8896a5d-a593-410b-9881-325f4bd51b59"). InnerVolumeSpecName "kube-api-access-xdn9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:56.833154 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.833122 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm" pod "a8896a5d-a593-410b-9881-325f4bd51b59" (UID: "a8896a5d-a593-410b-9881-325f4bd51b59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.834980 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.834954 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs" pod "a8896a5d-a593-410b-9881-325f4bd51b59" (UID: "a8896a5d-a593-410b-9881-325f4bd51b59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:56.932765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.932686 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8896a5d-a593-410b-9881-325f4bd51b59-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.932765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.932712 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8896a5d-a593-410b-9881-325f4bd51b59-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.932765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:56.932722 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdn9c\" (UniqueName: \"kubernetes.io/projected/a8896a5d-a593-410b-9881-325f4bd51b59-kube-api-access-xdn9c\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.650443 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.650373 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" Nov 02 17:56:57.650443 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.650416 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps" event={"ID":"a8896a5d-a593-410b-9881-325f4bd51b59","Type":"ContainerDied","Data":"4918800f0c8512bccac3e2f144c023d7be3c707811c448704368ad7cc7ec9adf"} Nov 02 17:56:57.650847 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.650473 6343 scope.go:122] "RemoveContainer" containerID="b8d4716326981824e11c8e7a34081393c3b43a66bd15e4a40c23eb70ab86fe70" Nov 02 17:56:57.650847 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.650581 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj" Nov 02 17:56:57.659489 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.659446 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" containerID="cri-o://ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163" gracePeriod=45 Nov 02 17:56:57.672519 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.672491 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" containerID="cri-o://9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3" gracePeriod=45 Nov 02 17:56:57.680629 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.680608 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj"] Nov 02 17:56:57.688368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.688348 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-w0-f37d16-kserve-dd98f4658-gj4pj"] Nov 02 17:56:57.692906 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.692881 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused" Nov 02 17:56:57.705077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.705041 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps"] Nov 02 17:56:57.707794 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.707777 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/distinct-2-w0-f37d16-kserve-f4fb84785-9zzps"] Nov 02 17:56:57.951218 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.951190 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:56:57.958344 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:57.958326 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:56:58.023184 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.023150 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" path="/var/lib/kubelet/pods/5f609015-e6bb-459a-b183-a242d0efcae1/volumes" Nov 02 17:56:58.023747 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.023725 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" path="/var/lib/kubelet/pods/a8896a5d-a593-410b-9881-325f4bd51b59/volumes" Nov 02 17:56:58.047798 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.047768 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache\") pod \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " Nov 02 17:56:58.047981 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.047821 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj\" (UniqueName: \"kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj\") pod \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " Nov 02 17:56:58.047981 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.047895 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm\") pod \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " Nov 02 17:56:58.048125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048105 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache" pod "3d1da72f-9b81-46ea-96d9-4f5cd4732415" (UID: "3d1da72f-9b81-46ea-96d9-4f5cd4732415"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.048178 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048149 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7\" (UniqueName: \"kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7\") pod \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " Nov 02 17:56:58.048226 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048196 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm\") pod \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " Nov 02 17:56:58.048280 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048262 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home\") pod \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " Nov 02 17:56:58.048337 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048294 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs\") pod \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " Nov 02 17:56:58.048337 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048330 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs\") pod \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " Nov 02 17:56:58.048437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048374 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache\") pod \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " Nov 02 17:56:58.048437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048420 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir\") pod \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " Nov 02 17:56:58.048542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048463 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home\") pod \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\" (UID: \"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a\") " Nov 02 17:56:58.048542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048492 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir\") pod \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\" (UID: \"3d1da72f-9b81-46ea-96d9-4f5cd4732415\") " Nov 02 17:56:58.048542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048524 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home" pod "3d1da72f-9b81-46ea-96d9-4f5cd4732415" (UID: "3d1da72f-9b81-46ea-96d9-4f5cd4732415"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.048757 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048736 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir" pod "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" (UID: "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.048817 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048753 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache" pod "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" (UID: "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.048986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.048963 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir" pod "3d1da72f-9b81-46ea-96d9-4f5cd4732415" (UID: "3d1da72f-9b81-46ea-96d9-4f5cd4732415"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.049176 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049156 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home" pod "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" (UID: "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.049356 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049337 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.049429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049362 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.049429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049376 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.049429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049389 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.049429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049403 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.049429 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.049415 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.050830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.050793 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7" pod "3d1da72f-9b81-46ea-96d9-4f5cd4732415" (UID: "3d1da72f-9b81-46ea-96d9-4f5cd4732415"). InnerVolumeSpecName "kube-api-access-k4kv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:58.050830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.050826 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj" pod "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" (UID: "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a"). InnerVolumeSpecName "kube-api-access-pfzrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:58.051243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.051221 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs" pod "3d1da72f-9b81-46ea-96d9-4f5cd4732415" (UID: "3d1da72f-9b81-46ea-96d9-4f5cd4732415"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:58.051391 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.051372 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs" pod "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" (UID: "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:58.051452 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.051392 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm" pod "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" (UID: "dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.052204 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.052181 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm" pod "3d1da72f-9b81-46ea-96d9-4f5cd4732415" (UID: "3d1da72f-9b81-46ea-96d9-4f5cd4732415"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.150897 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.150866 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfzrj\" (UniqueName: \"kubernetes.io/projected/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-kube-api-access-pfzrj\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.150897 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.150895 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1da72f-9b81-46ea-96d9-4f5cd4732415-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.151110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.150907 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4kv7\" (UniqueName: \"kubernetes.io/projected/3d1da72f-9b81-46ea-96d9-4f5cd4732415-kube-api-access-k4kv7\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.151110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.150918 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.151110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.150928 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.151110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.150947 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1da72f-9b81-46ea-96d9-4f5cd4732415-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.658455 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.658420 6343 generic.go:356] "Generic (PLEG): container finished" podID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerID="9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3" exitCode=0 Nov 02 17:56:58.658936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.658467 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" Nov 02 17:56:58.658936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.658507 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" event={"ID":"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a","Type":"ContainerDied","Data":"9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3"} Nov 02 17:56:58.658936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.658559 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2" event={"ID":"dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a","Type":"ContainerDied","Data":"879db3afd0f7164af5bef9c809e6ca6adf61c17b2264c4081b42954d9b1e70c0"} Nov 02 17:56:58.658936 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.658582 6343 scope.go:122] "RemoveContainer" containerID="9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3" Nov 02 17:56:58.660193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.660171 6343 generic.go:356] "Generic (PLEG): container finished" podID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerID="ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163" exitCode=0 Nov 02 17:56:58.660321 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.660208 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" Nov 02 17:56:58.660321 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.660261 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" event={"ID":"3d1da72f-9b81-46ea-96d9-4f5cd4732415","Type":"ContainerDied","Data":"ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163"} Nov 02 17:56:58.660321 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.660298 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf" event={"ID":"3d1da72f-9b81-46ea-96d9-4f5cd4732415","Type":"ContainerDied","Data":"0e0390f45681a68be4a9d5f1810a00b9bae1d884e45b6b19b7adcc3271c70bfb"} Nov 02 17:56:58.669566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.669542 6343 scope.go:122] "RemoveContainer" containerID="9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3" Nov 02 17:56:58.669883 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:58.669859 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3\": container with ID starting with 9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3 not found: ID does not exist" containerID="9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3" Nov 02 17:56:58.669966 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.669894 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3"} err="failed to get container status \"9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3\": rpc error: code = NotFound desc = could not find container \"9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3\": container with ID starting with 9b3c36260c0dbe183201778e91907c3d3c119cda538ec847e2f6fd16558c08d3 not found: ID does not exist" Nov 02 17:56:58.669966 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.669919 6343 scope.go:122] "RemoveContainer" containerID="ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163" Nov 02 17:56:58.678857 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.678833 6343 scope.go:122] "RemoveContainer" containerID="ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163" Nov 02 17:56:58.679146 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:58.679128 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163\": container with ID starting with ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163 not found: ID does not exist" containerID="ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163" Nov 02 17:56:58.679218 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.679156 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163"} err="failed to get container status \"ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163\": rpc error: code = NotFound desc = could not find container \"ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163\": container with ID starting with ef9701d3dc89651ddc5b0b581c1c4f5b686a2bdf47e3798e566850f500f2c163 not found: ID does not exist" Nov 02 17:56:58.688666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.688646 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2"] Nov 02 17:56:58.691286 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.691264 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/simulator-w5-ea3867-kserve-769c76b546-pfgd2"] Nov 02 17:56:58.701940 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.701919 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf"] Nov 02 17:56:58.707512 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:56:58.707494 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/premium-w5-ea3867-kserve-6f444bb759-4tdsf"] Nov 02 17:56:59.247264 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:56:59.247232 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:57:00.020532 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:00.020499 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" path="/var/lib/kubelet/pods/3d1da72f-9b81-46ea-96d9-4f5cd4732415/volumes" Nov 02 17:57:00.020952 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:00.020939 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" path="/var/lib/kubelet/pods/dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a/volumes" Nov 02 17:57:04.380908 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:04.380862 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:13.465528 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.465497 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:57:13.604802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.604771 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt\" (UniqueName: \"kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt\") pod \"06b7f858-3475-4ba6-ba7e-ac334712e015\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " Nov 02 17:57:13.604992 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.604971 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume\") pod \"06b7f858-3475-4ba6-ba7e-ac334712e015\" (UID: \"06b7f858-3475-4ba6-ba7e-ac334712e015\") " Nov 02 17:57:13.605402 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.605378 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume" pod "06b7f858-3475-4ba6-ba7e-ac334712e015" (UID: "06b7f858-3475-4ba6-ba7e-ac334712e015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:13.605557 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.605543 6343 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b7f858-3475-4ba6-ba7e-ac334712e015-plugins-config-volume\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:13.606897 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.606880 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt" pod "06b7f858-3475-4ba6-ba7e-ac334712e015" (UID: "06b7f858-3475-4ba6-ba7e-ac334712e015"). InnerVolumeSpecName "kube-api-access-695xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:13.707368 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.707294 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-695xt\" (UniqueName: \"kubernetes.io/projected/06b7f858-3475-4ba6-ba7e-ac334712e015-kube-api-access-695xt\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:13.785503 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.785464 6343 generic.go:356] "Generic (PLEG): container finished" podID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerID="67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0" exitCode=0 Nov 02 17:57:13.785718 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.785514 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" Nov 02 17:57:13.785718 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.785542 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" event={"ID":"06b7f858-3475-4ba6-ba7e-ac334712e015","Type":"ContainerDied","Data":"67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0"} Nov 02 17:57:13.785718 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.785635 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p" event={"ID":"06b7f858-3475-4ba6-ba7e-ac334712e015","Type":"ContainerDied","Data":"321aa48642aefc6174cea5ecdbd9c245b984ff4b01fc76c0f58a455bf70441c5"} Nov 02 17:57:13.785718 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.785667 6343 scope.go:122] "RemoveContainer" containerID="67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0" Nov 02 17:57:13.795611 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.795595 6343 scope.go:122] "RemoveContainer" containerID="67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0" Nov 02 17:57:13.795855 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:13.795833 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0\": container with ID starting with 67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0 not found: ID does not exist" containerID="67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0" Nov 02 17:57:13.795919 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.795861 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0"} err="failed to get container status \"67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0\": rpc error: code = NotFound desc = could not find container \"67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0\": container with ID starting with 67b2eaef054adfeee6259b8900484b076c8467907707eb1219eea53451668af0 not found: ID does not exist" Nov 02 17:57:13.813211 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.813188 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p"] Nov 02 17:57:13.815947 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:13.815928 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ea3867-6cfd4d7b65-49s7p"] Nov 02 17:57:14.022761 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:14.022678 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" path="/var/lib/kubelet/pods/06b7f858-3475-4ba6-ba7e-ac334712e015/volumes" Nov 02 17:57:18.053266 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:18.053183 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp"] Nov 02 17:57:18.054038 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:18.054002 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" containerID="cri-o://482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7" gracePeriod=30 Nov 02 17:57:18.101719 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:18.100994 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q"] Nov 02 17:57:18.101719 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:18.101233 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" containerID="cri-o://ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829" gracePeriod=30 Nov 02 17:57:18.132375 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:18.132331 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw"] Nov 02 17:57:18.139130 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:18.139091 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" containerID="cri-o://159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8" gracePeriod=30 Nov 02 17:57:23.440202 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.440174 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:57:23.446098 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.445555 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:57:23.526262 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.526231 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:57:23.563852 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.563818 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.563882 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.564129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.563933 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.564129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.563969 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.564129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564055 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.564129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564088 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564129 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564123 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564172 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564200 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564231 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564265 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564291 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564326 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564411 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564487 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.564955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564551 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.564955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.564614 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.567156 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567127 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.567278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567182 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.567278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567217 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.567278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567264 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.567447 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567293 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.567447 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567336 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.567447 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567412 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.567447 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567439 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.567662 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567487 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token\") pod \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\" (UID: \"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a\") " Nov 02 17:57:23.567662 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567529 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy\") pod \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\" (UID: \"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c\") " Nov 02 17:57:23.567662 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567582 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy\") pod \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\" (UID: \"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7\") " Nov 02 17:57:23.567826 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567799 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.568087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567885 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.568087 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.567901 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.568238 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568136 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.568385 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568363 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.568536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568517 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.568838 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568813 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.568956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568841 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.568956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568857 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.568956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568872 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.568956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568887 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.568956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568902 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.568956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568917 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.569479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.568951 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.569479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.569449 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.570123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.569818 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.570123 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.569897 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.570397 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.570362 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.570528 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.570505 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.570601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.570523 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.570753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.570720 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.571070 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.571028 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.571282 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.571259 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.574335 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.574311 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.574605 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.574582 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.576324 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.576302 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.576543 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.576523 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.577354 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.577335 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "kube-api-access-p77td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.577449 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.577360 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.577517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.577464 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "kube-api-access-prxnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.577517 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.577469 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy" pod "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" (UID: "0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.577625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.577587 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token" pod "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" (UID: "7d5e8853-02ff-4600-b3a4-2fcece5b2d6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.578679 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.578658 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt" pod "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" (UID: "1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a"). InnerVolumeSpecName "kube-api-access-zqtwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.671039 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.670949 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.671039 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.670981 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.671039 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.671041 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.671039 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.671074 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtwt\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-kube-api-access-zqtwt\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.671089 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.671998 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672016 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672029 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672088 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672103 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77td\" (UniqueName: \"kubernetes.io/projected/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-kube-api-access-p77td\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672117 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672132 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672146 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxnn\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-kube-api-access-prxnn\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672160 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672173 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672187 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672201 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672215 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672229 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.672278 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.672242 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.771171 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.771124 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg"] Nov 02 17:57:23.771445 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.771394 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" containerID="cri-o://b0e172753644f56f879557c14928772c8535f360b08f57ad5ac823a95d21260a" gracePeriod=30 Nov 02 17:57:23.775167 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.775107 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl"] Nov 02 17:57:23.775608 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.775500 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" containerID="cri-o://7d00bd4c097ffeb53a1cf5f6286571237ce66ba0315b7ad79621c7b2390ce520" gracePeriod=30 Nov 02 17:57:23.779099 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.779068 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l"] Nov 02 17:57:23.779340 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.779267 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" containerID="cri-o://4ffb9f2420a97512199d5b45dac0b8636fc4d054bef9ac0eeba9e307d667eb61" gracePeriod=30 Nov 02 17:57:23.857199 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.857166 6343 generic.go:356] "Generic (PLEG): container finished" podID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerID="482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7" exitCode=0 Nov 02 17:57:23.857381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.857225 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" Nov 02 17:57:23.857381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.857263 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" event={"ID":"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a","Type":"ContainerDied","Data":"482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7"} Nov 02 17:57:23.857381 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.857347 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp" event={"ID":"1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a","Type":"ContainerDied","Data":"f7be0a42971254cb8215e1c672999d9f480f8b52a0d5a5bbf17db6090b36f6e0"} Nov 02 17:57:23.857533 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.857388 6343 scope.go:122] "RemoveContainer" containerID="482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7" Nov 02 17:57:23.858931 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.858908 6343 generic.go:356] "Generic (PLEG): container finished" podID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerID="ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829" exitCode=0 Nov 02 17:57:23.859091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.858940 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" event={"ID":"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c","Type":"ContainerDied","Data":"ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829"} Nov 02 17:57:23.859091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.858964 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" Nov 02 17:57:23.859091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.858982 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q" event={"ID":"7d5e8853-02ff-4600-b3a4-2fcece5b2d6c","Type":"ContainerDied","Data":"d9e746c2280d46dc7d1e098a6aea2d518e264399021fa0ddcab028db7bf7cbf5"} Nov 02 17:57:23.860584 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.860560 6343 generic.go:356] "Generic (PLEG): container finished" podID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerID="159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8" exitCode=0 Nov 02 17:57:23.860685 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.860597 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" Nov 02 17:57:23.860685 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.860629 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" event={"ID":"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7","Type":"ContainerDied","Data":"159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8"} Nov 02 17:57:23.860685 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.860680 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw" event={"ID":"0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7","Type":"ContainerDied","Data":"81fb8e75d4110af8762114029cc1cd7cce176316a0a0e5bfba4c371fd5958d47"} Nov 02 17:57:23.871901 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.871852 6343 scope.go:122] "RemoveContainer" containerID="482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7" Nov 02 17:57:23.872297 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:23.872268 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7\": container with ID starting with 482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7 not found: ID does not exist" containerID="482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7" Nov 02 17:57:23.872400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.872310 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7"} err="failed to get container status \"482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7\": rpc error: code = NotFound desc = could not find container \"482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7\": container with ID starting with 482dc2f7c380e7a1db7d40014805b97cd3f4b9a56f1bc1f90f7677f7bfc244b7 not found: ID does not exist" Nov 02 17:57:23.872400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.872339 6343 scope.go:122] "RemoveContainer" containerID="ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829" Nov 02 17:57:23.904105 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.895609 6343 scope.go:122] "RemoveContainer" containerID="ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829" Nov 02 17:57:23.914028 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.913963 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp"] Nov 02 17:57:23.915689 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:23.915250 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829\": container with ID starting with ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829 not found: ID does not exist" containerID="ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829" Nov 02 17:57:23.915689 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.915293 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829"} err="failed to get container status \"ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829\": rpc error: code = NotFound desc = could not find container \"ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829\": container with ID starting with ca08e338338741ac6f0af0d506b5f552eb9c4a92a97356014bf80595709a7829 not found: ID does not exist" Nov 02 17:57:23.915689 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.915318 6343 scope.go:122] "RemoveContainer" containerID="159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8" Nov 02 17:57:23.916537 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.916512 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wzmdp"] Nov 02 17:57:23.928896 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.928664 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q"] Nov 02 17:57:23.936400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.936380 6343 scope.go:122] "RemoveContainer" containerID="159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8" Nov 02 17:57:23.936750 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:23.936730 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8\": container with ID starting with 159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8 not found: ID does not exist" containerID="159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8" Nov 02 17:57:23.936806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.936760 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8"} err="failed to get container status \"159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8\": rpc error: code = NotFound desc = could not find container \"159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8\": container with ID starting with 159691a036a8fe0519692062929dfea4de056a17cd613ca48dfd82fe744c59d8 not found: ID does not exist" Nov 02 17:57:23.937193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.937160 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-26j8q"] Nov 02 17:57:23.946811 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.946790 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw"] Nov 02 17:57:23.955172 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:23.955148 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-xn5jw"] Nov 02 17:57:24.021802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:24.021771 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" path="/var/lib/kubelet/pods/0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7/volumes" Nov 02 17:57:24.022610 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:24.022583 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" path="/var/lib/kubelet/pods/1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a/volumes" Nov 02 17:57:24.023542 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:24.023516 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" path="/var/lib/kubelet/pods/7d5e8853-02ff-4600-b3a4-2fcece5b2d6c/volumes" Nov 02 17:57:28.917613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:28.917576 6343 generic.go:356] "Generic (PLEG): container finished" podID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerID="b0e172753644f56f879557c14928772c8535f360b08f57ad5ac823a95d21260a" exitCode=0 Nov 02 17:57:28.918113 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:28.917677 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" event={"ID":"c325a4f3-27a3-49b7-a3da-911e936fc65f","Type":"ContainerDied","Data":"b0e172753644f56f879557c14928772c8535f360b08f57ad5ac823a95d21260a"} Nov 02 17:57:28.920345 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:28.920315 6343 generic.go:356] "Generic (PLEG): container finished" podID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerID="7d00bd4c097ffeb53a1cf5f6286571237ce66ba0315b7ad79621c7b2390ce520" exitCode=0 Nov 02 17:57:28.920706 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:28.920480 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" event={"ID":"3fa3af74-dd62-4b7c-a04c-64d32b90bc71","Type":"ContainerDied","Data":"7d00bd4c097ffeb53a1cf5f6286571237ce66ba0315b7ad79621c7b2390ce520"} Nov 02 17:57:28.922516 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:28.922488 6343 generic.go:356] "Generic (PLEG): container finished" podID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerID="4ffb9f2420a97512199d5b45dac0b8636fc4d054bef9ac0eeba9e307d667eb61" exitCode=0 Nov 02 17:57:28.922651 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:28.922540 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" event={"ID":"68ba9bc0-8708-4e4b-854d-c2eb5256260e","Type":"ContainerDied","Data":"4ffb9f2420a97512199d5b45dac0b8636fc4d054bef9ac0eeba9e307d667eb61"} Nov 02 17:57:29.150069 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.150026 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:57:29.196481 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.196454 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:57:29.198706 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.198684 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:57:29.275092 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:29.272803 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:57:29.307400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307365 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.307547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307473 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.307547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307510 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.307547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307540 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307574 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307606 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307635 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307670 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307724 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307756 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307787 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307800 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307882 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.308058 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.307928 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308138 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308153 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308190 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308218 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308274 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308322 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308344 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308355 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308422 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308430 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308590 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.308617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308607 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308641 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308658 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308703 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308736 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket\") pod \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\" (UID: \"3fa3af74-dd62-4b7c-a04c-64d32b90bc71\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308839 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308874 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308942 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket\") pod \"c325a4f3-27a3-49b7-a3da-911e936fc65f\" (UID: \"c325a4f3-27a3-49b7-a3da-911e936fc65f\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.308994 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv\") pod \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\" (UID: \"68ba9bc0-8708-4e4b-854d-c2eb5256260e\") " Nov 02 17:57:29.309209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309014 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.309729 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309706 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309796 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309737 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309796 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309753 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309796 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309767 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309796 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309781 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309796 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309809 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309823 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.309984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.309703 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.310237 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.310031 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.310778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.310435 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.310778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.310629 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.310778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.310676 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.310778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.310719 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.311065 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.310916 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.312193 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312172 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.312296 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312256 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.312380 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312343 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.312380 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312375 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.312498 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312388 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.312583 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312561 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.312783 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.312760 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "kube-api-access-rq9fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.313375 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.313350 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy" pod "c325a4f3-27a3-49b7-a3da-911e936fc65f" (UID: "c325a4f3-27a3-49b7-a3da-911e936fc65f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.313469 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.313407 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "kube-api-access-m2bcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.313599 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.313581 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token" pod "3fa3af74-dd62-4b7c-a04c-64d32b90bc71" (UID: "3fa3af74-dd62-4b7c-a04c-64d32b90bc71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.313787 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.313773 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "kube-api-access-h6qmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.314543 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.314522 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo" pod "68ba9bc0-8708-4e4b-854d-c2eb5256260e" (UID: "68ba9bc0-8708-4e4b-854d-c2eb5256260e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410723 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410752 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410763 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410772 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410782 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9fz\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-kube-api-access-rq9fz\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410790 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.410802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410798 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410809 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6qmv\" (UniqueName: \"kubernetes.io/projected/68ba9bc0-8708-4e4b-854d-c2eb5256260e-kube-api-access-h6qmv\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410818 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410826 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410834 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410844 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410852 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410861 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325a4f3-27a3-49b7-a3da-911e936fc65f-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410869 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410878 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410886 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325a4f3-27a3-49b7-a3da-911e936fc65f-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410894 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ba9bc0-8708-4e4b-854d-c2eb5256260e-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.411283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.410903 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2bcf\" (UniqueName: \"kubernetes.io/projected/3fa3af74-dd62-4b7c-a04c-64d32b90bc71-kube-api-access-m2bcf\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.930810 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.930780 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" Nov 02 17:57:29.931286 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.930809 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl" event={"ID":"3fa3af74-dd62-4b7c-a04c-64d32b90bc71","Type":"ContainerDied","Data":"67fec5cc39f57e0989d9ec45529c6c2bd8ad25c9f716a1999e9684c9a96448a6"} Nov 02 17:57:29.931286 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.930867 6343 scope.go:122] "RemoveContainer" containerID="7d00bd4c097ffeb53a1cf5f6286571237ce66ba0315b7ad79621c7b2390ce520" Nov 02 17:57:29.935115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.935094 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" Nov 02 17:57:29.935232 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.935094 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l" event={"ID":"68ba9bc0-8708-4e4b-854d-c2eb5256260e","Type":"ContainerDied","Data":"a5d3e7de22d84f42fb69b90938529f7edd6970450ecb880d95029021bbfb0ade"} Nov 02 17:57:29.936692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.936667 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" event={"ID":"c325a4f3-27a3-49b7-a3da-911e936fc65f","Type":"ContainerDied","Data":"f9086200199b98d772aa47b23812875a99cb3be8d01dae07cd057f5bc7aeb06f"} Nov 02 17:57:29.936692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.936685 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg" Nov 02 17:57:29.944986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.944968 6343 scope.go:122] "RemoveContainer" containerID="4ffb9f2420a97512199d5b45dac0b8636fc4d054bef9ac0eeba9e307d667eb61" Nov 02 17:57:29.954618 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.954603 6343 scope.go:122] "RemoveContainer" containerID="b0e172753644f56f879557c14928772c8535f360b08f57ad5ac823a95d21260a" Nov 02 17:57:29.969988 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.969961 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl"] Nov 02 17:57:29.984636 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:29.984614 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-6ttjl"] Nov 02 17:57:30.004524 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.004499 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg"] Nov 02 17:57:30.010949 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.010927 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-hgstg"] Nov 02 17:57:30.020233 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.020209 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" path="/var/lib/kubelet/pods/3fa3af74-dd62-4b7c-a04c-64d32b90bc71/volumes" Nov 02 17:57:30.020728 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.020710 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" path="/var/lib/kubelet/pods/c325a4f3-27a3-49b7-a3da-911e936fc65f/volumes" Nov 02 17:57:30.026547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.026526 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l"] Nov 02 17:57:30.036091 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.036070 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-qvr7l"] Nov 02 17:57:30.135108 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.135080 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh"] Nov 02 17:57:30.149637 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:30.149606 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d"] Nov 02 17:57:32.020576 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:32.020546 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" path="/var/lib/kubelet/pods/68ba9bc0-8708-4e4b-854d-c2eb5256260e/volumes" Nov 02 17:57:42.646601 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:42.646538 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg"] Nov 02 17:57:42.652716 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:42.652691 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k"] Nov 02 17:57:42.662173 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:42.662143 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2"] Nov 02 17:57:42.670265 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:42.670241 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r"] Nov 02 17:57:45.156939 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.156901 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" containerID="cri-o://b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c" gracePeriod=45 Nov 02 17:57:45.172533 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.172473 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" containerID="cri-o://621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee" gracePeriod=45 Nov 02 17:57:45.436181 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.436153 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:57:45.467823 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.467802 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:57:45.526909 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.526879 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir\") pod \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " Nov 02 17:57:45.527188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.526923 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs\") pod \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " Nov 02 17:57:45.527188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527062 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home\") pod \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " Nov 02 17:57:45.527188 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527120 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn\" (UniqueName: \"kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn\") pod \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " Nov 02 17:57:45.527394 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527267 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm\") pod \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " Nov 02 17:57:45.527394 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527368 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache\") pod \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\" (UID: \"5385c3d7-b069-49ce-9ffb-733703b7e3e3\") " Nov 02 17:57:45.527394 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527232 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir" pod "5385c3d7-b069-49ce-9ffb-733703b7e3e3" (UID: "5385c3d7-b069-49ce-9ffb-733703b7e3e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.527622 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527340 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home" pod "5385c3d7-b069-49ce-9ffb-733703b7e3e3" (UID: "5385c3d7-b069-49ce-9ffb-733703b7e3e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.527707 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.527666 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache" pod "5385c3d7-b069-49ce-9ffb-733703b7e3e3" (UID: "5385c3d7-b069-49ce-9ffb-733703b7e3e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.528440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.528419 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.528440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.528444 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.528440 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.528459 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.529870 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.529847 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm" pod "5385c3d7-b069-49ce-9ffb-733703b7e3e3" (UID: "5385c3d7-b069-49ce-9ffb-733703b7e3e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.530077 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.530021 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn" pod "5385c3d7-b069-49ce-9ffb-733703b7e3e3" (UID: "5385c3d7-b069-49ce-9ffb-733703b7e3e3"). InnerVolumeSpecName "kube-api-access-tjtsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:45.530145 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.530125 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs" pod "5385c3d7-b069-49ce-9ffb-733703b7e3e3" (UID: "5385c3d7-b069-49ce-9ffb-733703b7e3e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:45.629511 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629479 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs\") pod \"3a32b109-a7d9-4804-a76e-028a78fbffb1\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " Nov 02 17:57:45.629715 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629607 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm\") pod \"3a32b109-a7d9-4804-a76e-028a78fbffb1\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " Nov 02 17:57:45.629715 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629663 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home\") pod \"3a32b109-a7d9-4804-a76e-028a78fbffb1\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " Nov 02 17:57:45.629715 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629712 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache\") pod \"3a32b109-a7d9-4804-a76e-028a78fbffb1\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " Nov 02 17:57:45.629889 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629732 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn\" (UniqueName: \"kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn\") pod \"3a32b109-a7d9-4804-a76e-028a78fbffb1\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " Nov 02 17:57:45.629889 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629878 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir\") pod \"3a32b109-a7d9-4804-a76e-028a78fbffb1\" (UID: \"3a32b109-a7d9-4804-a76e-028a78fbffb1\") " Nov 02 17:57:45.629997 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629961 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache" pod "3a32b109-a7d9-4804-a76e-028a78fbffb1" (UID: "3a32b109-a7d9-4804-a76e-028a78fbffb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.629997 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.629977 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home" pod "3a32b109-a7d9-4804-a76e-028a78fbffb1" (UID: "3a32b109-a7d9-4804-a76e-028a78fbffb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.630157 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630129 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir" pod "3a32b109-a7d9-4804-a76e-028a78fbffb1" (UID: "3a32b109-a7d9-4804-a76e-028a78fbffb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.630528 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630505 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5385c3d7-b069-49ce-9ffb-733703b7e3e3-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.630617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630534 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.630617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630548 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5385c3d7-b069-49ce-9ffb-733703b7e3e3-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.630617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630560 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.630617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630573 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtsn\" (UniqueName: \"kubernetes.io/projected/5385c3d7-b069-49ce-9ffb-733703b7e3e3-kube-api-access-tjtsn\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.630617 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.630587 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.631580 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.631561 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs" pod "3a32b109-a7d9-4804-a76e-028a78fbffb1" (UID: "3a32b109-a7d9-4804-a76e-028a78fbffb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:45.631724 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.631706 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn" pod "3a32b109-a7d9-4804-a76e-028a78fbffb1" (UID: "3a32b109-a7d9-4804-a76e-028a78fbffb1"). InnerVolumeSpecName "kube-api-access-sdwvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:45.631769 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.631734 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm" pod "3a32b109-a7d9-4804-a76e-028a78fbffb1" (UID: "3a32b109-a7d9-4804-a76e-028a78fbffb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:45.732392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.732309 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdwvn\" (UniqueName: \"kubernetes.io/projected/3a32b109-a7d9-4804-a76e-028a78fbffb1-kube-api-access-sdwvn\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.732392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.732339 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a32b109-a7d9-4804-a76e-028a78fbffb1-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:45.732392 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:45.732350 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a32b109-a7d9-4804-a76e-028a78fbffb1-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:46.037409 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.037378 6343 generic.go:356] "Generic (PLEG): container finished" podID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerID="621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee" exitCode=0 Nov 02 17:57:46.037547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.037424 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" Nov 02 17:57:46.037547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.037459 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" event={"ID":"3a32b109-a7d9-4804-a76e-028a78fbffb1","Type":"ContainerDied","Data":"621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee"} Nov 02 17:57:46.037547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.037498 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d" event={"ID":"3a32b109-a7d9-4804-a76e-028a78fbffb1","Type":"ContainerDied","Data":"fc4e416ee806f9c24995cda07c4254ad546f839911c394e188fffc1b609a711f"} Nov 02 17:57:46.037547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.037519 6343 scope.go:122] "RemoveContainer" containerID="621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee" Nov 02 17:57:46.039427 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.039329 6343 generic.go:356] "Generic (PLEG): container finished" podID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerID="b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c" exitCode=0 Nov 02 17:57:46.039427 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.039371 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" event={"ID":"5385c3d7-b069-49ce-9ffb-733703b7e3e3","Type":"ContainerDied","Data":"b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c"} Nov 02 17:57:46.039427 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.039401 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" event={"ID":"5385c3d7-b069-49ce-9ffb-733703b7e3e3","Type":"ContainerDied","Data":"951fc8079785966e1f889eee2e66dfe2bc210fd765ea91a620d6e7d9d5cf8dd1"} Nov 02 17:57:46.039427 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.039414 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh" Nov 02 17:57:46.047296 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.047274 6343 scope.go:122] "RemoveContainer" containerID="621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee" Nov 02 17:57:46.047555 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:46.047535 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee\": container with ID starting with 621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee not found: ID does not exist" containerID="621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee" Nov 02 17:57:46.047598 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.047569 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee"} err="failed to get container status \"621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee\": rpc error: code = NotFound desc = could not find container \"621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee\": container with ID starting with 621f45ee2f96129fe12db123ade554cd1a429dea423c3d081a49373c8efd6cee not found: ID does not exist" Nov 02 17:57:46.047598 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.047589 6343 scope.go:122] "RemoveContainer" containerID="b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c" Nov 02 17:57:46.056841 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.056818 6343 scope.go:122] "RemoveContainer" containerID="b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c" Nov 02 17:57:46.057119 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:46.057102 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c\": container with ID starting with b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c not found: ID does not exist" containerID="b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c" Nov 02 17:57:46.057177 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.057134 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c"} err="failed to get container status \"b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c\": rpc error: code = NotFound desc = could not find container \"b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c\": container with ID starting with b2139ebbb0f0be370bf1bd99f40f4efb92f1f09ead27c8170914019936bfdb3c not found: ID does not exist" Nov 02 17:57:46.063242 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.063214 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh"] Nov 02 17:57:46.065384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.065364 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/distinct-2-w1-9dfa27-kserve-679b9c6768-5c4wh"] Nov 02 17:57:46.083236 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.083210 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d"] Nov 02 17:57:46.086642 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:46.086621 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9dfa27/premium-w1-9dfa27-kserve-74d999b4b4-8ck8d"] Nov 02 17:57:48.020751 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:48.020722 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" path="/var/lib/kubelet/pods/3a32b109-a7d9-4804-a76e-028a78fbffb1/volumes" Nov 02 17:57:48.021165 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:48.021150 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" path="/var/lib/kubelet/pods/5385c3d7-b069-49ce-9ffb-733703b7e3e3/volumes" Nov 02 17:57:55.498855 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.498819 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9"] Nov 02 17:57:55.499411 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.499092 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" podUID="1575541c-bc7c-4a50-a484-ac2f24abde57" containerName="maas-api" containerID="cri-o://5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08" gracePeriod=30 Nov 02 17:57:55.888325 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.888287 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:57:55.970798 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.970757 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs\") pod \"1575541c-bc7c-4a50-a484-ac2f24abde57\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " Nov 02 17:57:55.970993 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.970828 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx\" (UniqueName: \"kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx\") pod \"1575541c-bc7c-4a50-a484-ac2f24abde57\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " Nov 02 17:57:55.970993 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.970899 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls\") pod \"1575541c-bc7c-4a50-a484-ac2f24abde57\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " Nov 02 17:57:55.971128 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.971087 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca\") pod \"1575541c-bc7c-4a50-a484-ac2f24abde57\" (UID: \"1575541c-bc7c-4a50-a484-ac2f24abde57\") " Nov 02 17:57:55.972165 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.972136 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca" pod "1575541c-bc7c-4a50-a484-ac2f24abde57" (UID: "1575541c-bc7c-4a50-a484-ac2f24abde57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:55.974412 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.974385 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx" pod "1575541c-bc7c-4a50-a484-ac2f24abde57" (UID: "1575541c-bc7c-4a50-a484-ac2f24abde57"). InnerVolumeSpecName "kube-api-access-vnsxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:55.974492 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.974452 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs" pod "1575541c-bc7c-4a50-a484-ac2f24abde57" (UID: "1575541c-bc7c-4a50-a484-ac2f24abde57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:55.974605 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:55.974585 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls" pod "1575541c-bc7c-4a50-a484-ac2f24abde57" (UID: "1575541c-bc7c-4a50-a484-ac2f24abde57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:56.072919 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.072885 6343 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575541c-bc7c-4a50-a484-ac2f24abde57-openshift-service-ca\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:56.072919 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.072917 6343 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-metrics-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:56.073231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.072929 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsxx\" (UniqueName: \"kubernetes.io/projected/1575541c-bc7c-4a50-a484-ac2f24abde57-kube-api-access-vnsxx\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:56.073231 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.072941 6343 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575541c-bc7c-4a50-a484-ac2f24abde57-maas-api-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:56.104666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.104578 6343 generic.go:356] "Generic (PLEG): container finished" podID="1575541c-bc7c-4a50-a484-ac2f24abde57" containerID="5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08" exitCode=0 Nov 02 17:57:56.104666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.104628 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" event={"ID":"1575541c-bc7c-4a50-a484-ac2f24abde57","Type":"ContainerDied","Data":"5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08"} Nov 02 17:57:56.104666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.104666 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" event={"ID":"1575541c-bc7c-4a50-a484-ac2f24abde57","Type":"ContainerDied","Data":"27aeb7803fd46d121cc013685c9597d839660c56ac3e471474ed715eab98fe01"} Nov 02 17:57:56.104883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.104691 6343 scope.go:122] "RemoveContainer" containerID="5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08" Nov 02 17:57:56.104883 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.104827 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9" Nov 02 17:57:56.114452 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.114428 6343 scope.go:122] "RemoveContainer" containerID="5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08" Nov 02 17:57:56.114698 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:56.114676 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08\": container with ID starting with 5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08 not found: ID does not exist" containerID="5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08" Nov 02 17:57:56.114765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.114709 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08"} err="failed to get container status \"5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08\": rpc error: code = NotFound desc = could not find container \"5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08\": container with ID starting with 5d5128a7d931c6619ad9792829c384861380623ee1185fa6328315dfed5eae08 not found: ID does not exist" Nov 02 17:57:56.126243 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.126217 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9"] Nov 02 17:57:56.128582 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:56.128558 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9dfa27-8c7647d7-9f5r9"] Nov 02 17:57:57.669181 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:57.669138 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" containerID="cri-o://01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8" gracePeriod=45 Nov 02 17:57:57.675006 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:57.674977 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" containerID="cri-o://c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24" gracePeriod=45 Nov 02 17:57:57.685122 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:57.685092 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" containerID="cri-o://d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9" gracePeriod=45 Nov 02 17:57:57.692739 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:57.692678 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" containerID="cri-o://a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038" gracePeriod=45 Nov 02 17:57:57.973115 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:57.973083 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:57:57.989986 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:57.989955 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:57:58.006396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.006238 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:57:58.014004 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.013976 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:57:58.046076 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.045907 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1575541c-bc7c-4a50-a484-ac2f24abde57" path="/var/lib/kubelet/pods/1575541c-bc7c-4a50-a484-ac2f24abde57/volumes" Nov 02 17:57:58.096692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096660 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir\") pod \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " Nov 02 17:57:58.096692 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096697 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir\") pod \"95061dfc-67fd-434b-9ace-831026281d78\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096749 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home\") pod \"95061dfc-67fd-434b-9ace-831026281d78\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096771 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg\" (UniqueName: \"kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg\") pod \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096797 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir\") pod \"631fdc58-d4ce-495b-87e1-342fc876639d\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096819 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home\") pod \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096842 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm\") pod \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096888 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs\" (UniqueName: \"kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs\") pod \"631fdc58-d4ce-495b-87e1-342fc876639d\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096908 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm\") pod \"95061dfc-67fd-434b-9ace-831026281d78\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096937 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home\") pod \"631fdc58-d4ce-495b-87e1-342fc876639d\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096969 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache\") pod \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " Nov 02 17:57:58.096984 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.096991 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm\") pod \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097087 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj\" (UniqueName: \"kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj\") pod \"631fdc58-d4ce-495b-87e1-342fc876639d\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097107 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache\") pod \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097134 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir\") pod \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097158 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm\") pod \"631fdc58-d4ce-495b-87e1-342fc876639d\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097177 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache\") pod \"95061dfc-67fd-434b-9ace-831026281d78\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097208 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs\") pod \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\" (UID: \"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097285 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home\") pod \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097313 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk\" (UniqueName: \"kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk\") pod \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097358 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs\" (UniqueName: \"kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs\") pod \"95061dfc-67fd-434b-9ace-831026281d78\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097396 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs\") pod \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\" (UID: \"ddb8e678-a415-4baa-afc5-bb57cb3dc07d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097431 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache\") pod \"631fdc58-d4ce-495b-87e1-342fc876639d\" (UID: \"631fdc58-d4ce-495b-87e1-342fc876639d\") " Nov 02 17:57:58.097566 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.097452 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv\" (UniqueName: \"kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv\") pod \"95061dfc-67fd-434b-9ace-831026281d78\" (UID: \"95061dfc-67fd-434b-9ace-831026281d78\") " Nov 02 17:57:58.099613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.099338 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home" pod "ddb8e678-a415-4baa-afc5-bb57cb3dc07d" (UID: "ddb8e678-a415-4baa-afc5-bb57cb3dc07d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.099613 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.099406 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir" pod "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" (UID: "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.100082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.100014 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir" pod "95061dfc-67fd-434b-9ace-831026281d78" (UID: "95061dfc-67fd-434b-9ace-831026281d78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.100219 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.100190 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home" pod "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" (UID: "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.101569 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.101543 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache" pod "631fdc58-d4ce-495b-87e1-342fc876639d" (UID: "631fdc58-d4ce-495b-87e1-342fc876639d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.101711 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.101689 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir" pod "ddb8e678-a415-4baa-afc5-bb57cb3dc07d" (UID: "ddb8e678-a415-4baa-afc5-bb57cb3dc07d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.101949 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.101838 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home" pod "95061dfc-67fd-434b-9ace-831026281d78" (UID: "95061dfc-67fd-434b-9ace-831026281d78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.102545 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.102487 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache" pod "ddb8e678-a415-4baa-afc5-bb57cb3dc07d" (UID: "ddb8e678-a415-4baa-afc5-bb57cb3dc07d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.102678 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.102579 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache" pod "95061dfc-67fd-434b-9ace-831026281d78" (UID: "95061dfc-67fd-434b-9ace-831026281d78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.102961 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.102792 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm" pod "ddb8e678-a415-4baa-afc5-bb57cb3dc07d" (UID: "ddb8e678-a415-4baa-afc5-bb57cb3dc07d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.103125 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.103095 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home" pod "631fdc58-d4ce-495b-87e1-342fc876639d" (UID: "631fdc58-d4ce-495b-87e1-342fc876639d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.103405 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.103242 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir" pod "631fdc58-d4ce-495b-87e1-342fc876639d" (UID: "631fdc58-d4ce-495b-87e1-342fc876639d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.103650 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.103621 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache" pod "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" (UID: "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.104585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.104548 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm" pod "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" (UID: "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.104815 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.104778 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs" pod "631fdc58-d4ce-495b-87e1-342fc876639d" (UID: "631fdc58-d4ce-495b-87e1-342fc876639d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:58.105369 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.105338 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs" pod "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" (UID: "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:58.110666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110598 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm" pod "631fdc58-d4ce-495b-87e1-342fc876639d" (UID: "631fdc58-d4ce-495b-87e1-342fc876639d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.110666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110609 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm" pod "95061dfc-67fd-434b-9ace-831026281d78" (UID: "95061dfc-67fd-434b-9ace-831026281d78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:58.110666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110615 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs" pod "ddb8e678-a415-4baa-afc5-bb57cb3dc07d" (UID: "ddb8e678-a415-4baa-afc5-bb57cb3dc07d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:58.110666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110614 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj" pod "631fdc58-d4ce-495b-87e1-342fc876639d" (UID: "631fdc58-d4ce-495b-87e1-342fc876639d"). InnerVolumeSpecName "kube-api-access-shhcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:58.110666 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110661 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs" pod "95061dfc-67fd-434b-9ace-831026281d78" (UID: "95061dfc-67fd-434b-9ace-831026281d78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:58.110892 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110751 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv" pod "95061dfc-67fd-434b-9ace-831026281d78" (UID: "95061dfc-67fd-434b-9ace-831026281d78"). InnerVolumeSpecName "kube-api-access-frqrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:58.110892 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.110874 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk" pod "ddb8e678-a415-4baa-afc5-bb57cb3dc07d" (UID: "ddb8e678-a415-4baa-afc5-bb57cb3dc07d"). InnerVolumeSpecName "kube-api-access-t2jmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:58.111856 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.111836 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg" pod "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" (UID: "6079b1c1-3ab9-4de0-ad98-d98d27bc60e9"). InnerVolumeSpecName "kube-api-access-rm2sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:58.119467 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.119440 6343 generic.go:356] "Generic (PLEG): container finished" podID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerID="01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8" exitCode=0 Nov 02 17:57:58.119596 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.119507 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" Nov 02 17:57:58.119596 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.119524 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" event={"ID":"ddb8e678-a415-4baa-afc5-bb57cb3dc07d","Type":"ContainerDied","Data":"01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8"} Nov 02 17:57:58.119596 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.119574 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg" event={"ID":"ddb8e678-a415-4baa-afc5-bb57cb3dc07d","Type":"ContainerDied","Data":"9daa841b8e56bc248be12ff03947ffebaa9a3103c41772e120188e5ba09ef0c1"} Nov 02 17:57:58.119753 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.119596 6343 scope.go:122] "RemoveContainer" containerID="01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8" Nov 02 17:57:58.120967 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.120942 6343 generic.go:356] "Generic (PLEG): container finished" podID="95061dfc-67fd-434b-9ace-831026281d78" containerID="d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9" exitCode=0 Nov 02 17:57:58.121117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.120997 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" Nov 02 17:57:58.121117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.121064 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" event={"ID":"95061dfc-67fd-434b-9ace-831026281d78","Type":"ContainerDied","Data":"d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9"} Nov 02 17:57:58.121117 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.121093 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2" event={"ID":"95061dfc-67fd-434b-9ace-831026281d78","Type":"ContainerDied","Data":"40d2e23268d701999313ca487b4b99b7da1477aebaf0234bbb3b7ee3197df77d"} Nov 02 17:57:58.122641 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.122618 6343 generic.go:356] "Generic (PLEG): container finished" podID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerID="c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24" exitCode=0 Nov 02 17:57:58.122792 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.122776 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" event={"ID":"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9","Type":"ContainerDied","Data":"c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24"} Nov 02 17:57:58.122859 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.122808 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" event={"ID":"6079b1c1-3ab9-4de0-ad98-d98d27bc60e9","Type":"ContainerDied","Data":"5ea604af6961c543b1381a6b27ab24d37b5261985f4173fec95164ee5948b71d"} Nov 02 17:57:58.122914 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.122887 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k" Nov 02 17:57:58.129641 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.129621 6343 scope.go:122] "RemoveContainer" containerID="01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8" Nov 02 17:57:58.130015 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:58.129990 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8\": container with ID starting with 01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8 not found: ID does not exist" containerID="01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8" Nov 02 17:57:58.130110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.130038 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8"} err="failed to get container status \"01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8\": rpc error: code = NotFound desc = could not find container \"01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8\": container with ID starting with 01d6956b54622342bc3e05d36c1b44f4dafedc3aebc5b222827097eec72428d8 not found: ID does not exist" Nov 02 17:57:58.130110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.130092 6343 scope.go:122] "RemoveContainer" containerID="d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9" Nov 02 17:57:58.130351 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.130330 6343 generic.go:356] "Generic (PLEG): container finished" podID="631fdc58-d4ce-495b-87e1-342fc876639d" containerID="a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038" exitCode=0 Nov 02 17:57:58.130407 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.130384 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" Nov 02 17:57:58.130407 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.130397 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" event={"ID":"631fdc58-d4ce-495b-87e1-342fc876639d","Type":"ContainerDied","Data":"a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038"} Nov 02 17:57:58.130512 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.130425 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r" event={"ID":"631fdc58-d4ce-495b-87e1-342fc876639d","Type":"ContainerDied","Data":"584bf3b6a9960f5a7878d518567279fd909507932033fb753a6416ce60255b8a"} Nov 02 17:57:58.145181 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.145158 6343 scope.go:122] "RemoveContainer" containerID="d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9" Nov 02 17:57:58.145446 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:58.145421 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9\": container with ID starting with d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9 not found: ID does not exist" containerID="d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9" Nov 02 17:57:58.145531 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.145455 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9"} err="failed to get container status \"d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9\": rpc error: code = NotFound desc = could not find container \"d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9\": container with ID starting with d90c66b1029dd934b3c612b88b1a3f904c3c2a242b249d90b442c345a31a6ea9 not found: ID does not exist" Nov 02 17:57:58.145531 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.145480 6343 scope.go:122] "RemoveContainer" containerID="c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24" Nov 02 17:57:58.153506 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.153488 6343 scope.go:122] "RemoveContainer" containerID="c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24" Nov 02 17:57:58.153740 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:58.153723 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24\": container with ID starting with c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24 not found: ID does not exist" containerID="c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24" Nov 02 17:57:58.153780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.153749 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24"} err="failed to get container status \"c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24\": rpc error: code = NotFound desc = could not find container \"c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24\": container with ID starting with c2d9f1a62db74ac1b9504c8b409ca7a4cebcb567beafb6265dee12154cd52d24 not found: ID does not exist" Nov 02 17:57:58.153780 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.153766 6343 scope.go:122] "RemoveContainer" containerID="a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038" Nov 02 17:57:58.161557 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.161526 6343 scope.go:122] "RemoveContainer" containerID="a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038" Nov 02 17:57:58.161821 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:58.161802 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038\": container with ID starting with a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038 not found: ID does not exist" containerID="a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038" Nov 02 17:57:58.161899 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.161829 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038"} err="failed to get container status \"a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038\": rpc error: code = NotFound desc = could not find container \"a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038\": container with ID starting with a016a3549f4ebb59f79ad6edf904271052e810e7dea20cbb87c53e4fb2338038 not found: ID does not exist" Nov 02 17:57:58.163802 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.163778 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg"] Nov 02 17:57:58.173712 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.173684 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-2-w3-2a295c-kserve-6f8d54bd65-7kpbg"] Nov 02 17:57:58.185990 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.185966 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2"] Nov 02 17:57:58.197198 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.197166 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/premium-w3-2a295c-kserve-6c744f99b7-ltkv2"] Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199840 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199880 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199898 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhcj\" (UniqueName: \"kubernetes.io/projected/631fdc58-d4ce-495b-87e1-342fc876639d-kube-api-access-shhcj\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199915 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199929 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199942 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199966 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199979 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.199992 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200004 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jmk\" (UniqueName: \"kubernetes.io/projected/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-kube-api-access-t2jmk\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200017 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95061dfc-67fd-434b-9ace-831026281d78-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200029 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200041 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200091 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqrv\" (UniqueName: \"kubernetes.io/projected/95061dfc-67fd-434b-9ace-831026281d78-kube-api-access-frqrv\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200104 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddb8e678-a415-4baa-afc5-bb57cb3dc07d-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200116 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200128 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200140 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm2sg\" (UniqueName: \"kubernetes.io/projected/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-kube-api-access-rm2sg\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200152 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.200166 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.202126 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.202144 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631fdc58-d4ce-495b-87e1-342fc876639d-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.202157 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95061dfc-67fd-434b-9ace-831026281d78-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.204437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.202170 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631fdc58-d4ce-495b-87e1-342fc876639d-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:58.209114 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.209034 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k"] Nov 02 17:57:58.216819 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.216792 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/distinct-w3-2a295c-kserve-79f5b94658-nr48k"] Nov 02 17:57:58.229547 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.229511 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r"] Nov 02 17:57:58.231843 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:57:58.231816 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2a295c/simulator-w3-2a295c-kserve-7c6f6d676b-czd4r"] Nov 02 17:57:59.246776 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:57:59.246743 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:58:00.019892 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:00.019859 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" path="/var/lib/kubelet/pods/6079b1c1-3ab9-4de0-ad98-d98d27bc60e9/volumes" Nov 02 17:58:00.020380 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:00.020365 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" path="/var/lib/kubelet/pods/631fdc58-d4ce-495b-87e1-342fc876639d/volumes" Nov 02 17:58:00.020806 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:00.020792 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95061dfc-67fd-434b-9ace-831026281d78" path="/var/lib/kubelet/pods/95061dfc-67fd-434b-9ace-831026281d78/volumes" Nov 02 17:58:00.021565 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:00.021548 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" path="/var/lib/kubelet/pods/ddb8e678-a415-4baa-afc5-bb57cb3dc07d/volumes" Nov 02 17:58:05.351830 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:05.351793 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x"] Nov 02 17:58:05.352599 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:05.352541 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" containerID="cri-o://f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4" gracePeriod=30 Nov 02 17:58:05.368725 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:05.368693 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm"] Nov 02 17:58:05.368930 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:05.368908 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" containerID="cri-o://6062cde469c18de70628ee101113a038bd50731848e828649b2c0cb2d10e5aa6" gracePeriod=30 Nov 02 17:58:05.404618 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:05.404587 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh"] Nov 02 17:58:05.405185 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:05.405134 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" containerID="cri-o://0a0a0a009bcd5144d48d596e26288a6a6a4f45cc544d3bd0378f7a856e1312d9" gracePeriod=30 Nov 02 17:58:11.009306 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.009278 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:58:11.065519 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065467 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.065733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065540 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.065733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065640 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.065733 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065674 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.066001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065810 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.066001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065841 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.066001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065903 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.066001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065943 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.066001 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.065995 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token\") pod \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\" (UID: \"7c48ca71-e254-46fb-b6e5-f5f350caf7ab\") " Nov 02 17:58:11.073449 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.073258 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.076237 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.076168 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.078625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.078435 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.081459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.081437 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.120596 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.120516 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "kube-api-access-45rd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.127362 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.127243 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.139166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.138512 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.139166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.138666 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.139166 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.138712 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token" pod "7c48ca71-e254-46fb-b6e5-f5f350caf7ab" (UID: "7c48ca71-e254-46fb-b6e5-f5f350caf7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178493 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178532 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rd7\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-kube-api-access-45rd7\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178550 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178564 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178578 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178592 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178608 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178624 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.178664 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.178637 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48ca71-e254-46fb-b6e5-f5f350caf7ab-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.224862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.224360 6343 generic.go:356] "Generic (PLEG): container finished" podID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerID="f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4" exitCode=0 Nov 02 17:58:11.224862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.224461 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" event={"ID":"7c48ca71-e254-46fb-b6e5-f5f350caf7ab","Type":"ContainerDied","Data":"f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4"} Nov 02 17:58:11.224862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.224515 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" event={"ID":"7c48ca71-e254-46fb-b6e5-f5f350caf7ab","Type":"ContainerDied","Data":"4d20176cdbc90cfc202ea97e69a29173485f0cfc66910e915eed836056be7ae0"} Nov 02 17:58:11.224862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.224545 6343 scope.go:122] "RemoveContainer" containerID="f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4" Nov 02 17:58:11.224862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.224709 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x" Nov 02 17:58:11.265317 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.265290 6343 generic.go:356] "Generic (PLEG): container finished" podID="735322c1-4325-4305-9b43-4f56770ef13d" containerID="6062cde469c18de70628ee101113a038bd50731848e828649b2c0cb2d10e5aa6" exitCode=0 Nov 02 17:58:11.265532 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.265498 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" event={"ID":"735322c1-4325-4305-9b43-4f56770ef13d","Type":"ContainerDied","Data":"6062cde469c18de70628ee101113a038bd50731848e828649b2c0cb2d10e5aa6"} Nov 02 17:58:11.267209 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.267193 6343 generic.go:356] "Generic (PLEG): container finished" podID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerID="0a0a0a009bcd5144d48d596e26288a6a6a4f45cc544d3bd0378f7a856e1312d9" exitCode=0 Nov 02 17:58:11.267428 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.267415 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" event={"ID":"9844d25c-81ac-41b4-9e20-7dafd1a64058","Type":"ContainerDied","Data":"0a0a0a009bcd5144d48d596e26288a6a6a4f45cc544d3bd0378f7a856e1312d9"} Nov 02 17:58:11.293648 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.293213 6343 scope.go:122] "RemoveContainer" containerID="f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4" Nov 02 17:58:11.293648 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:11.293495 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4\": container with ID starting with f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4 not found: ID does not exist" containerID="f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4" Nov 02 17:58:11.293648 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.293520 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4"} err="failed to get container status \"f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4\": rpc error: code = NotFound desc = could not find container \"f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4\": container with ID starting with f16fc43160a906edf70c6d5e18f9ec88286804c4f5788df0c760ca41aa6e06e4 not found: ID does not exist" Nov 02 17:58:11.297606 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.297582 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x"] Nov 02 17:58:11.301855 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.301585 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-4z85x"] Nov 02 17:58:11.308711 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.308672 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:58:11.314358 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.314337 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:58:11.381126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381015 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381126 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381092 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381355 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381178 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381355 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381214 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381355 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381324 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381490 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381369 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381490 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381406 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381490 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381448 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381526 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381581 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381625 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381615 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381644 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381686 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381772 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381744 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.381925 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381784 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381925 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381833 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.381925 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381884 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data\") pod \"9844d25c-81ac-41b4-9e20-7dafd1a64058\" (UID: \"9844d25c-81ac-41b4-9e20-7dafd1a64058\") " Nov 02 17:58:11.382110 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.381972 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket\") pod \"735322c1-4325-4305-9b43-4f56770ef13d\" (UID: \"735322c1-4325-4305-9b43-4f56770ef13d\") " Nov 02 17:58:11.382199 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.382168 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.382503 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.382480 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.382751 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.382730 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.382925 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.382906 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.383124 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383107 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735322c1-4325-4305-9b43-4f56770ef13d-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.383191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383131 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.383191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383145 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.383191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383159 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.383191 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383104 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.383418 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383375 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.383905 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.383854 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.384152 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.384117 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.384309 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.384292 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.384478 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.384463 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.384608 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.384582 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.385481 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.385411 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.385862 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.385838 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.385945 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.385841 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.386017 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.385980 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "kube-api-access-bftkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.386017 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.385974 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy" pod "9844d25c-81ac-41b4-9e20-7dafd1a64058" (UID: "9844d25c-81ac-41b4-9e20-7dafd1a64058"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.387424 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.387398 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "kube-api-access-k8dkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.387860 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.387835 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token" pod "735322c1-4325-4305-9b43-4f56770ef13d" (UID: "735322c1-4325-4305-9b43-4f56770ef13d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.484540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484507 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484540 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484540 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8dkw\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-kube-api-access-k8dkw\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484554 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484567 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484580 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftkm\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-kube-api-access-bftkm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484593 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484605 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484617 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735322c1-4325-4305-9b43-4f56770ef13d-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484641 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9844d25c-81ac-41b4-9e20-7dafd1a64058-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484654 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735322c1-4325-4305-9b43-4f56770ef13d-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484668 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484679 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9844d25c-81ac-41b4-9e20-7dafd1a64058-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484691 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.484779 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:11.484703 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735322c1-4325-4305-9b43-4f56770ef13d-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:12.023383 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.021796 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" path="/var/lib/kubelet/pods/7c48ca71-e254-46fb-b6e5-f5f350caf7ab/volumes" Nov 02 17:58:12.275119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.275032 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" event={"ID":"735322c1-4325-4305-9b43-4f56770ef13d","Type":"ContainerDied","Data":"29eb00e21a306306b1c8463ef93e332cd827a2121907cbadd9f3b88ee5b2b96c"} Nov 02 17:58:12.275119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.275074 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm" Nov 02 17:58:12.275119 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.275096 6343 scope.go:122] "RemoveContainer" containerID="6062cde469c18de70628ee101113a038bd50731848e828649b2c0cb2d10e5aa6" Nov 02 17:58:12.276658 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.276641 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" Nov 02 17:58:12.276743 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.276699 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh" event={"ID":"9844d25c-81ac-41b4-9e20-7dafd1a64058","Type":"ContainerDied","Data":"c97dce1ca2f9d6c00e8fc26cc983a76975fb254d21fa63e3ee7056ac40434081"} Nov 02 17:58:12.284638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.284617 6343 scope.go:122] "RemoveContainer" containerID="0a0a0a009bcd5144d48d596e26288a6a6a4f45cc544d3bd0378f7a856e1312d9" Nov 02 17:58:12.300464 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.300439 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm"] Nov 02 17:58:12.306931 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.306912 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-p49hm"] Nov 02 17:58:12.319995 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.319972 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh"] Nov 02 17:58:12.327709 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:12.327687 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-6xcrh"] Nov 02 17:58:14.022017 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.021986 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735322c1-4325-4305-9b43-4f56770ef13d" path="/var/lib/kubelet/pods/735322c1-4325-4305-9b43-4f56770ef13d/volumes" Nov 02 17:58:14.022520 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.022495 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" path="/var/lib/kubelet/pods/9844d25c-81ac-41b4-9e20-7dafd1a64058/volumes" Nov 02 17:58:14.566387 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.566355 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5"] Nov 02 17:58:14.566639 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.566600 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" containerID="cri-o://7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe" gracePeriod=30 Nov 02 17:58:14.568212 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.568190 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k"] Nov 02 17:58:14.568396 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.568375 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" containerID="cri-o://57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9" gracePeriod=30 Nov 02 17:58:14.612782 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.612743 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv"] Nov 02 17:58:14.613032 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:14.613001 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" containerID="cri-o://4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6" gracePeriod=30 Nov 02 17:58:15.549536 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:15.549495 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:58:15.549995 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:15.549561 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:58:17.587695 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:17.587658 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:58:17.588192 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:17.587722 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:58:17.938330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:17.938252 6343 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:58:17.938330 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:17.938309 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:58:19.886021 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.885980 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:58:19.903728 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.903701 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:58:19.942033 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.941988 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:58:19.994593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994561 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.994593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994598 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.994850 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994619 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.994850 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994655 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.994850 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994759 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.994850 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994820 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994852 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994884 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994888 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994929 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.994973 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995005 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995080 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995025 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:19.995413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995060 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995156 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995273 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.995413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995331 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.995413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995371 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data\") pod \"d20d8c67-8423-4a9c-be54-62a5fb389495\" (UID: \"d20d8c67-8423-4a9c-be54-62a5fb389495\") " Nov 02 17:58:19.995413 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995407 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.995633 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995462 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.995633 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.995551 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert\") pod \"1e89a018-3e27-4a43-ac81-d429849254a4\" (UID: \"1e89a018-3e27-4a43-ac81-d429849254a4\") " Nov 02 17:58:19.996163 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.996143 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20d8c67-8423-4a9c-be54-62a5fb389495-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:19.996215 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.996171 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:19.996215 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.996187 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:19.997459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.996699 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.997459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.996796 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.997459 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.997307 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.997809 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.997510 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.997809 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.997592 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.998956 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.998924 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:19.999498 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.999222 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:19.999600 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.999522 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.000038 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:19.999731 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.000180 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.000151 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:20.001229 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.001191 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.001673 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.001647 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "kube-api-access-jmmgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.001765 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.001707 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.002643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.002618 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl" pod "d20d8c67-8423-4a9c-be54-62a5fb389495" (UID: "d20d8c67-8423-4a9c-be54-62a5fb389495"). InnerVolumeSpecName "kube-api-access-xrwcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.003472 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.003447 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo" pod "1e89a018-3e27-4a43-ac81-d429849254a4" (UID: "1e89a018-3e27-4a43-ac81-d429849254a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.055746 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:20.055712 6343 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20d8c67_8423_4a9c_be54_62a5fb389495.slice/crio-5dbaec83faeeb66e9795546d6311422325f906dc184284d3eeb66aaf562fff0c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20d8c67_8423_4a9c_be54_62a5fb389495.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:58:20.097189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097099 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097189 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097175 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097433 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097246 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097433 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097305 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097433 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097361 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097440 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.097587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097447 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097528 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097587 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097557 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.097793 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097630 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs\") pod \"21914c59-5090-4c83-8605-f4801a52d8d0\" (UID: \"21914c59-5090-4c83-8605-f4801a52d8d0\") " Nov 02 17:58:20.098162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.097899 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:20.098162 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098085 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.098370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098159 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.098370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098173 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.098370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098335 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098351 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098370 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098365 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21914c59-5090-4c83-8605-f4801a52d8d0-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098379 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098392 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e89a018-3e27-4a43-ac81-d429849254a4-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098406 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098420 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwcl\" (UniqueName: \"kubernetes.io/projected/d20d8c67-8423-4a9c-be54-62a5fb389495-kube-api-access-xrwcl\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098434 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098446 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098459 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098474 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098489 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098502 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098516 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098532 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e89a018-3e27-4a43-ac81-d429849254a4-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098546 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098560 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e89a018-3e27-4a43-ac81-d429849254a4-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098575 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmgl\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-kube-api-access-jmmgl\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098589 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e89a018-3e27-4a43-ac81-d429849254a4-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.098643 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.098601 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20d8c67-8423-4a9c-be54-62a5fb389495-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.099690 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.099665 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.100082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.100033 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "kube-api-access-4fnbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.100082 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.100077 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.100185 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.100114 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy" pod "21914c59-5090-4c83-8605-f4801a52d8d0" (UID: "21914c59-5090-4c83-8605-f4801a52d8d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.199479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.199448 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21914c59-5090-4c83-8605-f4801a52d8d0-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.199479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.199475 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.199479 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.199485 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21914c59-5090-4c83-8605-f4801a52d8d0-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.199717 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.199495 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fnbm\" (UniqueName: \"kubernetes.io/projected/21914c59-5090-4c83-8605-f4801a52d8d0-kube-api-access-4fnbm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.333030 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.332925 6343 generic.go:356] "Generic (PLEG): container finished" podID="21914c59-5090-4c83-8605-f4801a52d8d0" containerID="4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6" exitCode=0 Nov 02 17:58:20.333249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.333071 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" event={"ID":"21914c59-5090-4c83-8605-f4801a52d8d0","Type":"ContainerDied","Data":"4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6"} Nov 02 17:58:20.333249 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.333168 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" event={"ID":"21914c59-5090-4c83-8605-f4801a52d8d0","Type":"ContainerDied","Data":"588cfbd94a16d12a02a1427790640540ae1effbf2cb15be45a443a13fd850bde"} Nov 02 17:58:20.335607 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.335575 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" event={"ID":"1e89a018-3e27-4a43-ac81-d429849254a4","Type":"ContainerDied","Data":"7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe"} Nov 02 17:58:20.335607 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.335408 6343 generic.go:356] "Generic (PLEG): container finished" podID="1e89a018-3e27-4a43-ac81-d429849254a4" containerID="7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe" exitCode=0 Nov 02 17:58:20.335899 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.333537 6343 scope.go:122] "RemoveContainer" containerID="4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6" Nov 02 17:58:20.335899 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.335514 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" Nov 02 17:58:20.335899 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.335790 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5" event={"ID":"1e89a018-3e27-4a43-ac81-d429849254a4","Type":"ContainerDied","Data":"afdecb8a655bed6807810d5facf7301728b18f78828525a055efd1ed2cb097d6"} Nov 02 17:58:20.337400 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.337069 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv" Nov 02 17:58:20.338620 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.338600 6343 generic.go:356] "Generic (PLEG): container finished" podID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerID="57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9" exitCode=0 Nov 02 17:58:20.338704 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.338660 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" event={"ID":"d20d8c67-8423-4a9c-be54-62a5fb389495","Type":"ContainerDied","Data":"57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9"} Nov 02 17:58:20.338704 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.338689 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" event={"ID":"d20d8c67-8423-4a9c-be54-62a5fb389495","Type":"ContainerDied","Data":"5dbaec83faeeb66e9795546d6311422325f906dc184284d3eeb66aaf562fff0c"} Nov 02 17:58:20.338790 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.338742 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k" Nov 02 17:58:20.355585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.354031 6343 scope.go:122] "RemoveContainer" containerID="4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6" Nov 02 17:58:20.355585 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:20.354781 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6\": container with ID starting with 4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6 not found: ID does not exist" containerID="4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6" Nov 02 17:58:20.355585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.354818 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6"} err="failed to get container status \"4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6\": rpc error: code = NotFound desc = could not find container \"4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6\": container with ID starting with 4f9a8d824ec6300b9097ebd03d2c62cd3b235f6203f2c51ce795c6f3596234a6 not found: ID does not exist" Nov 02 17:58:20.355585 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.354840 6343 scope.go:122] "RemoveContainer" containerID="7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe" Nov 02 17:58:20.364892 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.364866 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5"] Nov 02 17:58:20.366437 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.366419 6343 scope.go:122] "RemoveContainer" containerID="7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe" Nov 02 17:58:20.366741 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:20.366720 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe\": container with ID starting with 7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe not found: ID does not exist" containerID="7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe" Nov 02 17:58:20.366816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.366748 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe"} err="failed to get container status \"7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe\": rpc error: code = NotFound desc = could not find container \"7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe\": container with ID starting with 7c9d251b613071f2cbb5d2a3bf87771e71358a473c1c5a6ee0c05446ba56c1fe not found: ID does not exist" Nov 02 17:58:20.366816 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.366765 6343 scope.go:122] "RemoveContainer" containerID="57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9" Nov 02 17:58:20.370638 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.370619 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-2wzc5"] Nov 02 17:58:20.376460 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.376438 6343 scope.go:122] "RemoveContainer" containerID="57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9" Nov 02 17:58:20.376774 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:20.376753 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9\": container with ID starting with 57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9 not found: ID does not exist" containerID="57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9" Nov 02 17:58:20.376851 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.376783 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9"} err="failed to get container status \"57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9\": rpc error: code = NotFound desc = could not find container \"57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9\": container with ID starting with 57643b16bb487767abfd8941476db27a2361b78d194e2a89974d1f8f68750bb9 not found: ID does not exist" Nov 02 17:58:20.388271 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.388249 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv"] Nov 02 17:58:20.391825 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.391802 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-nklsv"] Nov 02 17:58:20.405550 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.405528 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k"] Nov 02 17:58:20.415439 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:20.415402 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-rb92k"] Nov 02 17:58:22.019873 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:22.019842 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" path="/var/lib/kubelet/pods/1e89a018-3e27-4a43-ac81-d429849254a4/volumes" Nov 02 17:58:22.020406 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:22.020379 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" path="/var/lib/kubelet/pods/21914c59-5090-4c83-8605-f4801a52d8d0/volumes" Nov 02 17:58:22.020965 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:22.020944 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" path="/var/lib/kubelet/pods/d20d8c67-8423-4a9c-be54-62a5fb389495/volumes" Nov 02 17:58:29.246346 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:29.246309 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:58:41.484950 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:58:41.484923 6343 scope.go:122] "RemoveContainer" containerID="7fc75bd04d119512a002f2dd3ab7624a5f07bc524735dd5f172878eb09bfb3ce" Nov 02 17:58:59.245346 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:58:59.245305 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:59:29.246226 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:59:29.246194 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 17:59:42.202255 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:42.202216 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t"] Nov 02 17:59:57.223568 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.223501 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" containerID="cri-o://f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d" gracePeriod=45 Nov 02 17:59:57.492955 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.492931 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:59:57.578917 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.578885 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir\") pod \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " Nov 02 17:59:57.579288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.578958 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs\") pod \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " Nov 02 17:59:57.579288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.578980 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm\") pod \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " Nov 02 17:59:57.579288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.579119 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache\") pod \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " Nov 02 17:59:57.579288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.579151 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home\") pod \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " Nov 02 17:59:57.580288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.579714 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home" pod "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" (UID: "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.580288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.579762 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir" pod "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" (UID: "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.580288 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.579936 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache" pod "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" (UID: "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.580607 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.580587 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.580672 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.580613 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.580672 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.580628 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.582079 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.582016 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm" pod "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" (UID: "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.582283 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.582127 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs" pod "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" (UID: "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:57.681462 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.681429 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp\" (UniqueName: \"kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp\") pod \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\" (UID: \"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a\") " Nov 02 17:59:57.681778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.681760 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.681778 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.681775 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.683384 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.683363 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp" pod "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" (UID: "7e6c74f3-c984-4f3e-90f7-525c66ee8a2a"). InnerVolumeSpecName "kube-api-access-vlqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:57.782410 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.782332 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlqpp\" (UniqueName: \"kubernetes.io/projected/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a-kube-api-access-vlqpp\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.868432 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.868402 6343 generic.go:356] "Generic (PLEG): container finished" podID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerID="f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d" exitCode=0 Nov 02 17:59:57.868593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.868449 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" Nov 02 17:59:57.868593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.868487 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" event={"ID":"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a","Type":"ContainerDied","Data":"f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d"} Nov 02 17:59:57.868593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.868535 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t" event={"ID":"7e6c74f3-c984-4f3e-90f7-525c66ee8a2a","Type":"ContainerDied","Data":"31f99a369a03d585fa03375428aa4d69580d66fb25b4b4aa6a1feabee15c9bad"} Nov 02 17:59:57.868593 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.868558 6343 scope.go:122] "RemoveContainer" containerID="f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d" Nov 02 17:59:57.878335 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.878317 6343 scope.go:122] "RemoveContainer" containerID="f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d" Nov 02 17:59:57.878589 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:59:57.878570 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d\": container with ID starting with f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d not found: ID does not exist" containerID="f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d" Nov 02 17:59:57.878631 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.878600 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d"} err="failed to get container status \"f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d\": rpc error: code = NotFound desc = could not find container \"f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d\": container with ID starting with f5dce1884afc3cd5526e1a1587a2187061ac15b11c8f6cbf5d744be7abc4581d not found: ID does not exist" Nov 02 17:59:57.892280 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.892256 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t"] Nov 02 17:59:57.896490 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:57.896471 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/simulator-w4-ea752f-kserve-86cf5c445-t667t"] Nov 02 17:59:58.019382 ip-10-0-141-84 kubenswrapper[6343]: I1102 17:59:58.019352 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" path="/var/lib/kubelet/pods/7e6c74f3-c984-4f3e-90f7-525c66ee8a2a/volumes" Nov 02 17:59:59.246329 ip-10-0-141-84 kubenswrapper[6343]: E1102 17:59:59.246296 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:00:03.571071 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:03.570391 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw"] Nov 02 18:00:03.571071 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:03.570726 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" podUID="c0dbad12-dc7f-4931-afd7-50492a5d8358" containerName="maas-api" containerID="cri-o://a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f" gracePeriod=30 Nov 02 18:00:03.966102 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:03.966065 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 18:00:04.010451 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.010416 6343 generic.go:356] "Generic (PLEG): container finished" podID="c0dbad12-dc7f-4931-afd7-50492a5d8358" containerID="a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f" exitCode=0 Nov 02 18:00:04.010617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.010445 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" event={"ID":"c0dbad12-dc7f-4931-afd7-50492a5d8358","Type":"ContainerDied","Data":"a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f"} Nov 02 18:00:04.010617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.010471 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" Nov 02 18:00:04.010617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.010507 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw" event={"ID":"c0dbad12-dc7f-4931-afd7-50492a5d8358","Type":"ContainerDied","Data":"229032452a83b16b9dc26e5235f1a8f54a95d06c3e80c45eb2385699b451b6f5"} Nov 02 18:00:04.010617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.010537 6343 scope.go:122] "RemoveContainer" containerID="a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f" Nov 02 18:00:04.021609 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.021585 6343 scope.go:122] "RemoveContainer" containerID="a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f" Nov 02 18:00:04.021942 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:00:04.021917 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f\": container with ID starting with a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f not found: ID does not exist" containerID="a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f" Nov 02 18:00:04.022055 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.021950 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f"} err="failed to get container status \"a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f\": rpc error: code = NotFound desc = could not find container \"a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f\": container with ID starting with a8a36f90055c80f7a474d7566b8aec6a33a7c6eb5ae91e6fc9e352be70ec235f not found: ID does not exist" Nov 02 18:00:04.111985 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.111949 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88\" (UniqueName: \"kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88\") pod \"c0dbad12-dc7f-4931-afd7-50492a5d8358\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " Nov 02 18:00:04.112161 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.112015 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs\") pod \"c0dbad12-dc7f-4931-afd7-50492a5d8358\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " Nov 02 18:00:04.112161 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.112077 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca\") pod \"c0dbad12-dc7f-4931-afd7-50492a5d8358\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " Nov 02 18:00:04.112161 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.112124 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls\") pod \"c0dbad12-dc7f-4931-afd7-50492a5d8358\" (UID: \"c0dbad12-dc7f-4931-afd7-50492a5d8358\") " Nov 02 18:00:04.112553 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.112513 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca" pod "c0dbad12-dc7f-4931-afd7-50492a5d8358" (UID: "c0dbad12-dc7f-4931-afd7-50492a5d8358"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:04.112717 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.112696 6343 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0dbad12-dc7f-4931-afd7-50492a5d8358-openshift-service-ca\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:04.114196 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.114173 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88" pod "c0dbad12-dc7f-4931-afd7-50492a5d8358" (UID: "c0dbad12-dc7f-4931-afd7-50492a5d8358"). InnerVolumeSpecName "kube-api-access-dbz88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:04.114292 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.114276 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs" pod "c0dbad12-dc7f-4931-afd7-50492a5d8358" (UID: "c0dbad12-dc7f-4931-afd7-50492a5d8358"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:00:04.114381 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.114353 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls" pod "c0dbad12-dc7f-4931-afd7-50492a5d8358" (UID: "c0dbad12-dc7f-4931-afd7-50492a5d8358"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:00:04.213846 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.213788 6343 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-maas-api-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:04.213846 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.213813 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbz88\" (UniqueName: \"kubernetes.io/projected/c0dbad12-dc7f-4931-afd7-50492a5d8358-kube-api-access-dbz88\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:04.213846 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.213823 6343 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0dbad12-dc7f-4931-afd7-50492a5d8358-metrics-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:04.335508 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.335479 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw"] Nov 02 18:00:04.340191 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:04.340166 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea752f-6cc5979bc5-ldbmw"] Nov 02 18:00:06.020177 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:06.020143 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0dbad12-dc7f-4931-afd7-50492a5d8358" path="/var/lib/kubelet/pods/c0dbad12-dc7f-4931-afd7-50492a5d8358/volumes" Nov 02 18:00:12.467616 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:12.467581 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb"] Nov 02 18:00:12.468141 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:12.467830 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" containerID="cri-o://45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785" gracePeriod=30 Nov 02 18:00:12.480930 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:12.480895 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h"] Nov 02 18:00:12.481207 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:12.481136 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" containerID="cri-o://4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41" gracePeriod=30 Nov 02 18:00:12.488073 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:12.487918 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv"] Nov 02 18:00:12.488184 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:12.488151 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" containerID="cri-o://f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5" gracePeriod=30 Nov 02 18:00:14.472321 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:14.472286 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:14.472695 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:14.472355 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:15.628098 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:15.628065 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:15.628466 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:15.628125 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:15.812133 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:15.812099 6343 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:15.812295 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:15.812151 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:17.756726 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.756655 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 18:00:17.804508 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.804480 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 18:00:17.807818 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.807793 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 18:00:17.855764 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855730 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.855923 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855784 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.855923 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855828 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.855923 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855884 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.855923 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855914 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.856198 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855940 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.856198 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.855986 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.856198 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856021 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.856198 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856075 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.856198 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856102 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.856198 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856139 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856206 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856240 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856278 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856311 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856373 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856402 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856432 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856471 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856485 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.856510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856505 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert\") pod \"0bc4e599-6084-4310-a694-8ed1e3d95777\" (UID: \"0bc4e599-6084-4310-a694-8ed1e3d95777\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856505 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856538 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856572 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856611 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856644 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data\") pod \"0bb0ff1b-6414-4938-a094-3d0626b7c845\" (UID: \"0bb0ff1b-6414-4938-a094-3d0626b7c845\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856672 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856722 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856748 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.857033 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.856782 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs\") pod \"ca212799-029c-4ded-a676-ed706a80533c\" (UID: \"ca212799-029c-4ded-a676-ed706a80533c\") " Nov 02 18:00:17.857521 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857041 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:17.857521 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857282 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.857521 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857347 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.857759 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857737 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.857823 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857765 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.857823 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857780 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.857823 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857793 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0ff1b-6414-4938-a094-3d0626b7c845-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.857823 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.857807 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca212799-029c-4ded-a676-ed706a80533c-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.858225 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.858201 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:17.858225 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.858444 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.858225 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.858608 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.858225 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.858793 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859234 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859014 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859234 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859103 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859452 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859424 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859524 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859496 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859612 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859553 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859953 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859919 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.859953 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.859933 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.860165 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.860075 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:17.860522 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.860399 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.860522 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.860475 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.860739 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.860565 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "kube-api-access-bjmzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.861655 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.861631 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token" pod "0bc4e599-6084-4310-a694-8ed1e3d95777" (UID: "0bc4e599-6084-4310-a694-8ed1e3d95777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.862209 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.862190 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:17.862368 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.862350 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g" pod "ca212799-029c-4ded-a676-ed706a80533c" (UID: "ca212799-029c-4ded-a676-ed706a80533c"). InnerVolumeSpecName "kube-api-access-7jg5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.862528 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.862510 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:17.863140 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.863117 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.863775 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.863748 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4" pod "0bb0ff1b-6414-4938-a094-3d0626b7c845" (UID: "0bb0ff1b-6414-4938-a094-3d0626b7c845"). InnerVolumeSpecName "kube-api-access-lndw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:17.958724 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958692 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc4e599-6084-4310-a694-8ed1e3d95777-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958727 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958743 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958757 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958770 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958784 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca212799-029c-4ded-a676-ed706a80533c-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958799 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958812 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958825 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.958848 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958840 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmzw\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-kube-api-access-bjmzw\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958857 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958870 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0ff1b-6414-4938-a094-3d0626b7c845-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958885 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958898 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958912 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958926 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958940 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc4e599-6084-4310-a694-8ed1e3d95777-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958955 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg5g\" (UniqueName: \"kubernetes.io/projected/ca212799-029c-4ded-a676-ed706a80533c-kube-api-access-7jg5g\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958968 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0ff1b-6414-4938-a094-3d0626b7c845-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958981 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.958994 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca212799-029c-4ded-a676-ed706a80533c-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.959142 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:17.959007 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lndw4\" (UniqueName: \"kubernetes.io/projected/0bb0ff1b-6414-4938-a094-3d0626b7c845-kube-api-access-lndw4\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.088097 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.088038 6343 generic.go:356] "Generic (PLEG): container finished" podID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerID="f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5" exitCode=0 Nov 02 18:00:18.088097 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.088084 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" event={"ID":"0bc4e599-6084-4310-a694-8ed1e3d95777","Type":"ContainerDied","Data":"f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5"} Nov 02 18:00:18.088280 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.088120 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" Nov 02 18:00:18.088280 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.088166 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv" event={"ID":"0bc4e599-6084-4310-a694-8ed1e3d95777","Type":"ContainerDied","Data":"df59f7d56df7c309a821a231b51cca553b926150680c4d8058a11b6f6339a5a0"} Nov 02 18:00:18.088280 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.088193 6343 scope.go:122] "RemoveContainer" containerID="f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5" Nov 02 18:00:18.089510 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.089482 6343 generic.go:356] "Generic (PLEG): container finished" podID="ca212799-029c-4ded-a676-ed706a80533c" containerID="4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41" exitCode=0 Nov 02 18:00:18.089623 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.089532 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" Nov 02 18:00:18.089623 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.089577 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" event={"ID":"ca212799-029c-4ded-a676-ed706a80533c","Type":"ContainerDied","Data":"4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41"} Nov 02 18:00:18.089732 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.089627 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h" event={"ID":"ca212799-029c-4ded-a676-ed706a80533c","Type":"ContainerDied","Data":"ab9b2a89d6c4f2b556f22f762058eb931e631220b069f90c75cdbd9b03666b04"} Nov 02 18:00:18.091229 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.091209 6343 generic.go:356] "Generic (PLEG): container finished" podID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerID="45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785" exitCode=0 Nov 02 18:00:18.091336 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.091276 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" Nov 02 18:00:18.091336 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.091293 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" event={"ID":"0bb0ff1b-6414-4938-a094-3d0626b7c845","Type":"ContainerDied","Data":"45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785"} Nov 02 18:00:18.091449 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.091338 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb" event={"ID":"0bb0ff1b-6414-4938-a094-3d0626b7c845","Type":"ContainerDied","Data":"ec781799fcd7b0f7ea926dc92336e22ba876cf1bbc1ce5180b17e1a3bc483211"} Nov 02 18:00:18.097791 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.097774 6343 scope.go:122] "RemoveContainer" containerID="f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5" Nov 02 18:00:18.098076 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:00:18.098040 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5\": container with ID starting with f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5 not found: ID does not exist" containerID="f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5" Nov 02 18:00:18.098162 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.098081 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5"} err="failed to get container status \"f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5\": rpc error: code = NotFound desc = could not find container \"f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5\": container with ID starting with f4a3aab99548443759e53642ea40299e4d7158ff659cdae75cc76120413fefe5 not found: ID does not exist" Nov 02 18:00:18.098162 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.098097 6343 scope.go:122] "RemoveContainer" containerID="4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41" Nov 02 18:00:18.106211 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.106168 6343 scope.go:122] "RemoveContainer" containerID="4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41" Nov 02 18:00:18.106395 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:00:18.106379 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41\": container with ID starting with 4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41 not found: ID does not exist" containerID="4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41" Nov 02 18:00:18.106456 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.106400 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41"} err="failed to get container status \"4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41\": rpc error: code = NotFound desc = could not find container \"4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41\": container with ID starting with 4693e419a5eb392a4f9f7f327e94908568b794606df99e29870f9c99716b5f41 not found: ID does not exist" Nov 02 18:00:18.106456 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.106413 6343 scope.go:122] "RemoveContainer" containerID="45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785" Nov 02 18:00:18.116349 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.116318 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv"] Nov 02 18:00:18.117459 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.117413 6343 scope.go:122] "RemoveContainer" containerID="45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785" Nov 02 18:00:18.117911 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:00:18.117888 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785\": container with ID starting with 45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785 not found: ID does not exist" containerID="45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785" Nov 02 18:00:18.118017 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.117921 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785"} err="failed to get container status \"45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785\": rpc error: code = NotFound desc = could not find container \"45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785\": container with ID starting with 45b6ef882a437fbab8d6ced0367b089a2cbd926407e34959c7b7ecae27876785 not found: ID does not exist" Nov 02 18:00:18.124793 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.124771 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-8vxzv"] Nov 02 18:00:18.143732 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.143704 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h"] Nov 02 18:00:18.152656 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.152627 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-w2j2h"] Nov 02 18:00:18.165853 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.165825 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb"] Nov 02 18:00:18.171070 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:18.171033 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-4dwlb"] Nov 02 18:00:20.021032 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:20.020998 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" path="/var/lib/kubelet/pods/0bb0ff1b-6414-4938-a094-3d0626b7c845/volumes" Nov 02 18:00:20.021535 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:20.021519 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" path="/var/lib/kubelet/pods/0bc4e599-6084-4310-a694-8ed1e3d95777/volumes" Nov 02 18:00:20.022075 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:20.022042 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca212799-029c-4ded-a676-ed706a80533c" path="/var/lib/kubelet/pods/ca212799-029c-4ded-a676-ed706a80533c/volumes" Nov 02 18:00:29.245671 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:00:29.245636 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:00:40.191819 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:40.191790 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:00:40.196426 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:00:40.196407 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:00:59.245515 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:00:59.245477 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:01:08.324525 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:08.324490 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn"] Nov 02 18:01:08.324932 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:08.324701 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" containerID="cri-o://f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c" gracePeriod=30 Nov 02 18:01:08.936606 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:08.936575 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v"] Nov 02 18:01:13.580328 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.580302 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 18:01:13.735381 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735284 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735381 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735385 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735406 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735438 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735479 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735510 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735535 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735559 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735625 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735598 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:13.735996 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735642 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:13.735996 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735653 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket\") pod \"15f46651-12fd-4b03-b004-b44d4af7daa5\" (UID: \"15f46651-12fd-4b03-b004-b44d4af7daa5\") " Nov 02 18:01:13.735996 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735755 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:13.736196 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.735995 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:13.736196 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.736130 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:13.736196 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.736186 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.736346 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.736209 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.736346 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.736224 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.736346 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.736237 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f46651-12fd-4b03-b004-b44d4af7daa5-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.737773 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.737748 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:13.737926 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.737905 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "kube-api-access-5gg9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:13.737926 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.737913 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:13.738005 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.737956 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo" pod "15f46651-12fd-4b03-b004-b44d4af7daa5" (UID: "15f46651-12fd-4b03-b004-b44d4af7daa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:13.837729 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.837688 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.837729 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.837721 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.837729 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.837731 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f46651-12fd-4b03-b004-b44d4af7daa5-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.837729 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.837741 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gg9z\" (UniqueName: \"kubernetes.io/projected/15f46651-12fd-4b03-b004-b44d4af7daa5-kube-api-access-5gg9z\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:13.837998 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:13.837752 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f46651-12fd-4b03-b004-b44d4af7daa5-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:14.375807 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.375770 6343 generic.go:356] "Generic (PLEG): container finished" podID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerID="f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c" exitCode=0 Nov 02 18:01:14.376085 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.375823 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" Nov 02 18:01:14.376085 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.375864 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" event={"ID":"15f46651-12fd-4b03-b004-b44d4af7daa5","Type":"ContainerDied","Data":"f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c"} Nov 02 18:01:14.376085 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.375990 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn" event={"ID":"15f46651-12fd-4b03-b004-b44d4af7daa5","Type":"ContainerDied","Data":"aa0fbad962a308e608431325b90f53d6d82ae67e5da678232fa8d31099fe4966"} Nov 02 18:01:14.376085 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.376030 6343 scope.go:122] "RemoveContainer" containerID="f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c" Nov 02 18:01:14.385165 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.385141 6343 scope.go:122] "RemoveContainer" containerID="f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c" Nov 02 18:01:14.385428 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:01:14.385408 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c\": container with ID starting with f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c not found: ID does not exist" containerID="f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c" Nov 02 18:01:14.385493 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.385436 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c"} err="failed to get container status \"f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c\": rpc error: code = NotFound desc = could not find container \"f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c\": container with ID starting with f5fda1f9f5c1552e84f7b51faa8e03be0b1854a204f84b8d08763ef96b793c2c not found: ID does not exist" Nov 02 18:01:14.400025 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.399993 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn"] Nov 02 18:01:14.411129 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:14.411100 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-q2xpn"] Nov 02 18:01:16.019061 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:16.019018 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" path="/var/lib/kubelet/pods/15f46651-12fd-4b03-b004-b44d4af7daa5/volumes" Nov 02 18:01:23.957708 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:23.957661 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" containerID="cri-o://ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84" gracePeriod=45 Nov 02 18:01:24.209748 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.209695 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 18:01:24.243700 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.243656 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm\") pod \"f6eadac3-d485-445e-8edd-107faff8c296\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " Nov 02 18:01:24.243868 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.243742 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs\") pod \"f6eadac3-d485-445e-8edd-107faff8c296\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " Nov 02 18:01:24.244237 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.244197 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache\") pod \"f6eadac3-d485-445e-8edd-107faff8c296\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " Nov 02 18:01:24.244404 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.244239 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home\") pod \"f6eadac3-d485-445e-8edd-107faff8c296\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " Nov 02 18:01:24.244404 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.244289 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml\" (UniqueName: \"kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml\") pod \"f6eadac3-d485-445e-8edd-107faff8c296\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " Nov 02 18:01:24.245635 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.245605 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir\") pod \"f6eadac3-d485-445e-8edd-107faff8c296\" (UID: \"f6eadac3-d485-445e-8edd-107faff8c296\") " Nov 02 18:01:24.246286 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.246243 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir" pod "f6eadac3-d485-445e-8edd-107faff8c296" (UID: "f6eadac3-d485-445e-8edd-107faff8c296"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.246450 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.246350 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache" pod "f6eadac3-d485-445e-8edd-107faff8c296" (UID: "f6eadac3-d485-445e-8edd-107faff8c296"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.247028 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.247000 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home" pod "f6eadac3-d485-445e-8edd-107faff8c296" (UID: "f6eadac3-d485-445e-8edd-107faff8c296"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.248908 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.248882 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs" pod "f6eadac3-d485-445e-8edd-107faff8c296" (UID: "f6eadac3-d485-445e-8edd-107faff8c296"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:24.249007 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.248982 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml" pod "f6eadac3-d485-445e-8edd-107faff8c296" (UID: "f6eadac3-d485-445e-8edd-107faff8c296"). InnerVolumeSpecName "kube-api-access-b4mml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:24.250115 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.250081 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm" pod "f6eadac3-d485-445e-8edd-107faff8c296" (UID: "f6eadac3-d485-445e-8edd-107faff8c296"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.347201 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.347171 6343 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-model-cache\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.347201 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.347196 6343 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-home\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.347364 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.347209 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mml\" (UniqueName: \"kubernetes.io/projected/f6eadac3-d485-445e-8edd-107faff8c296-kube-api-access-b4mml\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.347364 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.347239 6343 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-tmp-dir\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.347364 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.347248 6343 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6eadac3-d485-445e-8edd-107faff8c296-dshm\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.347364 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.347256 6343 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6eadac3-d485-445e-8edd-107faff8c296-tls-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.425741 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.425712 6343 generic.go:356] "Generic (PLEG): container finished" podID="f6eadac3-d485-445e-8edd-107faff8c296" containerID="ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84" exitCode=0 Nov 02 18:01:24.425894 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.425758 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" Nov 02 18:01:24.425894 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.425805 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" event={"ID":"f6eadac3-d485-445e-8edd-107faff8c296","Type":"ContainerDied","Data":"ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84"} Nov 02 18:01:24.425894 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.425840 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v" event={"ID":"f6eadac3-d485-445e-8edd-107faff8c296","Type":"ContainerDied","Data":"1a1094a0ef7fee89d184932a088b2d4a4fd2ae43aa1851985437c404e0414715"} Nov 02 18:01:24.425894 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.425858 6343 scope.go:122] "RemoveContainer" containerID="ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84" Nov 02 18:01:24.434694 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.434677 6343 scope.go:122] "RemoveContainer" containerID="ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84" Nov 02 18:01:24.434977 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:01:24.434960 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84\": container with ID starting with ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84 not found: ID does not exist" containerID="ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84" Nov 02 18:01:24.435026 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.434986 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84"} err="failed to get container status \"ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84\": rpc error: code = NotFound desc = could not find container \"ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84\": container with ID starting with ac1d6b36a142391a9e8937dade7f5ba3ba9057a283f0b3240f075418bfd85f84 not found: ID does not exist" Nov 02 18:01:24.451440 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.451419 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v"] Nov 02 18:01:24.454889 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:24.454869 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/simulator-w2-bb4ecb-kserve-dd56d8959-vbw4v"] Nov 02 18:01:26.018860 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:26.018827 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6eadac3-d485-445e-8edd-107faff8c296" path="/var/lib/kubelet/pods/f6eadac3-d485-445e-8edd-107faff8c296/volumes" Nov 02 18:01:29.245298 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:01:29.245266 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:01:37.331283 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:37.331239 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2"] Nov 02 18:01:37.331733 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:37.331487 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" containerID="cri-o://b80174bf8db1a15397df4499c0d46d48c16923dcba250eae090c5843c22516f5" gracePeriod=30 Nov 02 18:01:37.332900 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:37.332879 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv"] Nov 02 18:01:37.333113 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:37.333078 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" containerID="cri-o://c00438e99f4ebf4bee55f3d0e2f8ee050b05643103912697a77bf9c97854ca2f" gracePeriod=30 Nov 02 18:01:42.521665 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:42.521635 6343 generic.go:356] "Generic (PLEG): container finished" podID="0b6b986a-922e-440d-948c-e82d40bdb696" containerID="b80174bf8db1a15397df4499c0d46d48c16923dcba250eae090c5843c22516f5" exitCode=0 Nov 02 18:01:42.522113 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:42.521711 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" event={"ID":"0b6b986a-922e-440d-948c-e82d40bdb696","Type":"ContainerDied","Data":"b80174bf8db1a15397df4499c0d46d48c16923dcba250eae090c5843c22516f5"} Nov 02 18:01:42.523519 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:42.523494 6343 generic.go:356] "Generic (PLEG): container finished" podID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerID="c00438e99f4ebf4bee55f3d0e2f8ee050b05643103912697a77bf9c97854ca2f" exitCode=0 Nov 02 18:01:42.523647 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:42.523585 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" event={"ID":"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a","Type":"ContainerDied","Data":"c00438e99f4ebf4bee55f3d0e2f8ee050b05643103912697a77bf9c97854ca2f"} Nov 02 18:01:42.925380 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:42.925246 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 18:01:43.022013 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.021985 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 18:01:43.034441 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034410 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034450 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034484 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034510 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034790 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034619 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034790 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034656 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034790 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034682 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034790 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034723 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.034790 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034757 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.034790 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.034780 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45\") pod \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\" (UID: \"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a\") " Nov 02 18:01:43.035171 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.035123 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.035171 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.035155 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.035279 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.035192 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.035279 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.035203 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:43.035437 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.035308 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.074746 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.074694 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:43.078302 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.078175 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.078302 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.078269 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:43.098627 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.098588 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45" pod "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" (UID: "b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a"). InnerVolumeSpecName "kube-api-access-jpd45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:43.136018 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.135994 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136172 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136097 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136172 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136131 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136298 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136180 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136298 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136240 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136298 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136287 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136455 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136345 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136455 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136375 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.136562 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136461 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket\") pod \"0b6b986a-922e-440d-948c-e82d40bdb696\" (UID: \"0b6b986a-922e-440d-948c-e82d40bdb696\") " Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136821 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpd45\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-kube-api-access-jpd45\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136845 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136862 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136877 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136890 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136903 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136915 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.136929 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.150230 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.167115 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.171460 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.167309 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.176469 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.176343 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.176469 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.176412 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.185253 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.183153 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "kube-api-access-mpfnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:43.185253 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.185205 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.188468 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.188435 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:43.200225 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.200184 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo" pod "0b6b986a-922e-440d-948c-e82d40bdb696" (UID: "0b6b986a-922e-440d-948c-e82d40bdb696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241588 6343 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241621 6343 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6b986a-922e-440d-948c-e82d40bdb696-istiod-ca-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241638 6343 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6b986a-922e-440d-948c-e82d40bdb696-istio-podinfo\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241653 6343 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-credential-socket\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241667 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241681 6343 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-data\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241696 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfnh\" (UniqueName: \"kubernetes.io/projected/0b6b986a-922e-440d-948c-e82d40bdb696-kube-api-access-mpfnh\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241712 6343 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-istio-envoy\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.258066 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.241726 6343 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6b986a-922e-440d-948c-e82d40bdb696-workload-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.530567 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.530493 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" Nov 02 18:01:43.531017 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.530493 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2" event={"ID":"0b6b986a-922e-440d-948c-e82d40bdb696","Type":"ContainerDied","Data":"811483d682747b2a84851759455e154c41740b1ee5ecb63d8a2d05da5b76e8ab"} Nov 02 18:01:43.531017 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.530626 6343 scope.go:122] "RemoveContainer" containerID="b80174bf8db1a15397df4499c0d46d48c16923dcba250eae090c5843c22516f5" Nov 02 18:01:43.532035 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.532015 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" event={"ID":"b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a","Type":"ContainerDied","Data":"972c205e746a816125a8889887effc57a6346ae0453e6d20f2d96b41d0c98eb8"} Nov 02 18:01:43.532153 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.532037 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv" Nov 02 18:01:43.574386 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.574174 6343 scope.go:122] "RemoveContainer" containerID="c00438e99f4ebf4bee55f3d0e2f8ee050b05643103912697a77bf9c97854ca2f" Nov 02 18:01:43.614093 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.603648 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2"] Nov 02 18:01:43.614093 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.612688 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s9gf2"] Nov 02 18:01:43.669427 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.668759 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv"] Nov 02 18:01:43.669427 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:43.668811 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-glljv"] Nov 02 18:01:44.022719 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:44.022679 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" path="/var/lib/kubelet/pods/0b6b986a-922e-440d-948c-e82d40bdb696/volumes" Nov 02 18:01:44.023361 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:01:44.023335 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" path="/var/lib/kubelet/pods/b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a/volumes" Nov 02 18:01:59.244921 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:01:59.244883 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:02:29.245135 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:02:29.245097 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:02:41.886525 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:41.886445 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx"] Nov 02 18:02:41.886917 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:41.886643 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" podUID="b647ac63-1ff8-43d5-a24f-ce773264ff27" containerName="discovery" containerID="cri-o://b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23" gracePeriod=30 Nov 02 18:02:42.135834 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.135809 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 18:02:42.318311 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318223 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318311 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318269 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318311 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318289 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318599 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318324 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318599 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318397 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318599 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318441 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318599 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318472 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert\") pod \"b647ac63-1ff8-43d5-a24f-ce773264ff27\" (UID: \"b647ac63-1ff8-43d5-a24f-ce773264ff27\") " Nov 02 18:02:42.318818 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.318703 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:42.320624 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.320571 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "kube-api-access-fwd9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:42.320750 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.320673 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:42.320975 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.320945 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:42.321115 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.321006 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:42.321115 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.321026 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:42.321115 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.321028 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token" pod "b647ac63-1ff8-43d5-a24f-ce773264ff27" (UID: "b647ac63-1ff8-43d5-a24f-ce773264ff27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:42.419378 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419349 6343 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-token\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.419378 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419373 6343 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-cacerts\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.419582 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419389 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwd9k\" (UniqueName: \"kubernetes.io/projected/b647ac63-1ff8-43d5-a24f-ce773264ff27-kube-api-access-fwd9k\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.419582 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419400 6343 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-dns-cert\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.419582 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419409 6343 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-csr-ca-configmap\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.419582 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419420 6343 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b647ac63-1ff8-43d5-a24f-ce773264ff27-local-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.419582 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.419428 6343 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b647ac63-1ff8-43d5-a24f-ce773264ff27-istio-kubeconfig\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:42.799540 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.799505 6343 generic.go:356] "Generic (PLEG): container finished" podID="b647ac63-1ff8-43d5-a24f-ce773264ff27" containerID="b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23" exitCode=0 Nov 02 18:02:42.799712 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.799552 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" Nov 02 18:02:42.799712 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.799587 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" event={"ID":"b647ac63-1ff8-43d5-a24f-ce773264ff27","Type":"ContainerDied","Data":"b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23"} Nov 02 18:02:42.799712 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.799638 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx" event={"ID":"b647ac63-1ff8-43d5-a24f-ce773264ff27","Type":"ContainerDied","Data":"af86c216e640055d34e0100ca6f81a17856884ef3e2e1fa48cf12da9962603b2"} Nov 02 18:02:42.799712 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.799664 6343 scope.go:122] "RemoveContainer" containerID="b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23" Nov 02 18:02:42.809156 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.809138 6343 scope.go:122] "RemoveContainer" containerID="b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23" Nov 02 18:02:42.809434 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:02:42.809411 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23\": container with ID starting with b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23 not found: ID does not exist" containerID="b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23" Nov 02 18:02:42.809482 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.809448 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23"} err="failed to get container status \"b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23\": rpc error: code = NotFound desc = could not find container \"b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23\": container with ID starting with b1e8eb60294320905ff1fba123d9e998109ba72c0250b7857d61922bfa4e1b23 not found: ID does not exist" Nov 02 18:02:42.821073 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.821029 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx"] Nov 02 18:02:42.826654 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:42.826627 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s42mx"] Nov 02 18:02:44.024074 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:02:44.024027 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b647ac63-1ff8-43d5-a24f-ce773264ff27" path="/var/lib/kubelet/pods/b647ac63-1ff8-43d5-a24f-ce773264ff27/volumes" Nov 02 18:02:59.246140 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:02:59.246104 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:03:29.257294 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:03:29.257200 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:03:59.245208 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:03:59.245172 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:04:29.247217 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:04:29.247171 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:04:59.244886 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:04:59.244848 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:05:29.317292 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:05:29.317254 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:05:40.221502 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:05:40.221474 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:05:40.227583 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:05:40.227560 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:05:59.244880 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:05:59.244848 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:06:29.244206 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:06:29.244132 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:06:59.244993 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:06:59.244958 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:07:29.245619 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:07:29.245585 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:07:59.245684 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:07:59.245647 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:08:29.246149 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:08:29.246108 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:08:59.245303 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:08:59.245264 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:09:29.244828 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:09:29.244792 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:09:59.245169 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:09:59.245134 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:10:29.244616 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:10:29.244575 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:10:40.325549 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:10:40.325512 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:10:40.329520 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:10:40.329499 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:10:57.851288 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:10:57.851198 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:57.851748 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:10:57.851274 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:59.245808 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:10:59.245772 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:11:29.245078 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:11:29.245017 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:11:59.248576 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:11:59.248536 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:12:29.245516 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:12:29.245442 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:12:59.246150 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:12:59.246110 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:13:29.249734 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:13:29.249696 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:13:46.068263 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:13:46.068215 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nc62c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:13:46.068881 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:13:46.068293 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nc62c" podUID="72508624-36f4-4d4a-992b-37271137a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:13:59.245269 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:13:59.245231 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:14:13.088759 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.088727 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb"] Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089115 6343 cpu_manager.go:414] "containerMap: removing container" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089128 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089141 6343 cpu_manager.go:414] "containerMap: removing container" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089146 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089154 6343 cpu_manager.go:414] "containerMap: removing container" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089158 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089166 6343 cpu_manager.go:414] "containerMap: removing container" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089171 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089178 6343 cpu_manager.go:414] "containerMap: removing container" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089184 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089191 6343 cpu_manager.go:414] "containerMap: removing container" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089196 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089203 6343 cpu_manager.go:414] "containerMap: removing container" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089207 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089214 6343 cpu_manager.go:414] "containerMap: removing container" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089218 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089223 6343 cpu_manager.go:414] "containerMap: removing container" podUID="b647ac63-1ff8-43d5-a24f-ce773264ff27" containerName="discovery" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089227 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b647ac63-1ff8-43d5-a24f-ce773264ff27" containerName="discovery" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089237 6343 cpu_manager.go:414] "containerMap: removing container" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089242 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089249 6343 cpu_manager.go:414] "containerMap: removing container" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089253 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" Nov 02 18:14:13.089250 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089262 6343 cpu_manager.go:414] "containerMap: removing container" podUID="c0dbad12-dc7f-4931-afd7-50492a5d8358" containerName="maas-api" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089266 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0dbad12-dc7f-4931-afd7-50492a5d8358" containerName="maas-api" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089272 6343 cpu_manager.go:414] "containerMap: removing container" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089276 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089282 6343 cpu_manager.go:414] "containerMap: removing container" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089286 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089292 6343 cpu_manager.go:414] "containerMap: removing container" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089296 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089302 6343 cpu_manager.go:414] "containerMap: removing container" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089305 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089312 6343 cpu_manager.go:414] "containerMap: removing container" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089316 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089323 6343 cpu_manager.go:414] "containerMap: removing container" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089327 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089334 6343 cpu_manager.go:414] "containerMap: removing container" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089338 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089344 6343 cpu_manager.go:414] "containerMap: removing container" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089348 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089359 6343 cpu_manager.go:414] "containerMap: removing container" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089363 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089368 6343 cpu_manager.go:414] "containerMap: removing container" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089372 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089379 6343 cpu_manager.go:414] "containerMap: removing container" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089383 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089393 6343 cpu_manager.go:414] "containerMap: removing container" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089396 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089403 6343 cpu_manager.go:414] "containerMap: removing container" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089407 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089413 6343 cpu_manager.go:414] "containerMap: removing container" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089417 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089424 6343 cpu_manager.go:414] "containerMap: removing container" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089428 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089434 6343 cpu_manager.go:414] "containerMap: removing container" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089439 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089449 6343 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089454 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089461 6343 cpu_manager.go:414] "containerMap: removing container" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089465 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089472 6343 cpu_manager.go:414] "containerMap: removing container" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089476 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089483 6343 cpu_manager.go:414] "containerMap: removing container" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089488 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089496 6343 cpu_manager.go:414] "containerMap: removing container" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089500 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089506 6343 cpu_manager.go:414] "containerMap: removing container" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089510 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089516 6343 cpu_manager.go:414] "containerMap: removing container" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089520 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089526 6343 cpu_manager.go:414] "containerMap: removing container" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089530 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089537 6343 cpu_manager.go:414] "containerMap: removing container" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089541 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089547 6343 cpu_manager.go:414] "containerMap: removing container" podUID="1575541c-bc7c-4a50-a484-ac2f24abde57" containerName="maas-api" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089552 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1575541c-bc7c-4a50-a484-ac2f24abde57" containerName="maas-api" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089559 6343 cpu_manager.go:414] "containerMap: removing container" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerName="payload-processing" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089563 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerName="payload-processing" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089621 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="631fdc58-d4ce-495b-87e1-342fc876639d" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089635 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="1575541c-bc7c-4a50-a484-ac2f24abde57" containerName="maas-api" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089643 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b7f858-3475-4ba6-ba7e-ac334712e015" containerName="payload-processing" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089654 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f609015-e6bb-459a-b183-a242d0efcae1" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089662 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="5385c3d7-b069-49ce-9ffb-733703b7e3e3" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089672 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd61ad35-435a-4acd-bb1b-9dd8f09e5cd5" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089676 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e89a018-3e27-4a43-ac81-d429849254a4" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089683 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f46651-12fd-4b03-b004-b44d4af7daa5" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089687 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc9c67d-05c1-433f-adc3-be3569128b55" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089692 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ba9bc0-8708-4e4b-854d-c2eb5256260e" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089697 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="9844d25c-81ac-41b4-9e20-7dafd1a64058" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089703 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="b647ac63-1ff8-43d5-a24f-ce773264ff27" containerName="discovery" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089708 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5e8853-02ff-4600-b3a4-2fcece5b2d6c" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089713 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="6916fd93-159d-47f9-bdfa-d87459f8d514" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089718 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0dbad12-dc7f-4931-afd7-50492a5d8358" containerName="maas-api" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089723 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6c74f3-c984-4f3e-90f7-525c66ee8a2a" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089727 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="95061dfc-67fd-434b-9ace-831026281d78" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089733 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c48ca71-e254-46fb-b6e5-f5f350caf7ab" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089737 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="21914c59-5090-4c83-8605-f4801a52d8d0" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089742 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="735322c1-4325-4305-9b43-4f56770ef13d" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089747 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdf4b07-d0e8-4f27-8c2d-f546992b6f5a" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089751 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc4e599-6084-4310-a694-8ed1e3d95777" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089756 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa3af74-dd62-4b7c-a04c-64d32b90bc71" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089762 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3f3ba8-94c1-45d0-a2ea-7fd8421b96d7" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089767 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb8e678-a415-4baa-afc5-bb57cb3dc07d" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089771 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1da72f-9b81-46ea-96d9-4f5cd4732415" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089775 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20d8c67-8423-4a9c-be54-62a5fb389495" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089781 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="6079b1c1-3ab9-4de0-ad98-d98d27bc60e9" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089785 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8896a5d-a593-410b-9881-325f4bd51b59" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089790 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7f7261-02e4-40a3-b507-112ce6ce4da4" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089794 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="c325a4f3-27a3-49b7-a3da-911e936fc65f" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089800 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a14e5-f0a7-44fe-ad6c-7c2fbe169e6a" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089805 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9f10d9-20b8-404c-bdeb-a64cc32d355f" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089810 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fc7ad3-63dd-4834-9ef9-3a7b20cbec7a" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089817 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6eadac3-d485-445e-8edd-107faff8c296" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089821 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a32b109-a7d9-4804-a76e-028a78fbffb1" containerName="main" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089826 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca212799-029c-4ded-a676-ed706a80533c" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089831 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb0ff1b-6414-4938-a094-3d0626b7c845" containerName="istio-proxy" Nov 02 18:14:13.089911 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.089836 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6b986a-922e-440d-948c-e82d40bdb696" containerName="istio-proxy" Nov 02 18:14:13.093190 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.092658 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.105499 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.105475 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb"] Nov 02 18:14:13.249719 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.249690 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5mbr\" (UniqueName: \"kubernetes.io/projected/4a7dd848-560f-4f65-83eb-1a4a7873290d-kube-api-access-k5mbr\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.249893 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.249725 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7dd848-560f-4f65-83eb-1a4a7873290d-metrics-certs\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.249893 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.249875 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7dd848-560f-4f65-83eb-1a4a7873290d-openshift-service-ca\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.249967 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.249945 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7dd848-560f-4f65-83eb-1a4a7873290d-maas-api-tls\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.350862 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.350768 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7dd848-560f-4f65-83eb-1a4a7873290d-maas-api-tls\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.351067 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.350892 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5mbr\" (UniqueName: \"kubernetes.io/projected/4a7dd848-560f-4f65-83eb-1a4a7873290d-kube-api-access-k5mbr\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.351159 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.351083 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7dd848-560f-4f65-83eb-1a4a7873290d-metrics-certs\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.351226 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.351184 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7dd848-560f-4f65-83eb-1a4a7873290d-openshift-service-ca\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.351855 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.351826 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7dd848-560f-4f65-83eb-1a4a7873290d-openshift-service-ca\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.353577 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.353549 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a7dd848-560f-4f65-83eb-1a4a7873290d-maas-api-tls\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.354060 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.354026 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a7dd848-560f-4f65-83eb-1a4a7873290d-metrics-certs\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.363167 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.363146 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5mbr\" (UniqueName: \"kubernetes.io/projected/4a7dd848-560f-4f65-83eb-1a4a7873290d-kube-api-access-k5mbr\") pod \"maas-api-5fd8dffdf6-c7nwb\" (UID: \"4a7dd848-560f-4f65-83eb-1a4a7873290d\") " pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.406430 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.406406 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:13.719041 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.719019 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb"] Nov 02 18:14:13.721035 ip-10-0-141-84 kubenswrapper[6343]: W1102 18:14:13.721005 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7dd848_560f_4f65_83eb_1a4a7873290d.slice/crio-52e73f8a4624e99a219a941c722c8333823e6a2e7925f4227fcc12c4a15e6de3 WatchSource:0}: Error finding container 52e73f8a4624e99a219a941c722c8333823e6a2e7925f4227fcc12c4a15e6de3: Status 404 returned error can't find the container with id 52e73f8a4624e99a219a941c722c8333823e6a2e7925f4227fcc12c4a15e6de3 Nov 02 18:14:13.722320 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:13.722302 6343 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:14:14.324281 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:14.324247 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" event={"ID":"4a7dd848-560f-4f65-83eb-1a4a7873290d","Type":"ContainerStarted","Data":"24c13b3fc8db04a687e14b0f1ca265721245b3af2d411623ad68db6fe35158e4"} Nov 02 18:14:14.324281 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:14.324303 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" event={"ID":"4a7dd848-560f-4f65-83eb-1a4a7873290d","Type":"ContainerStarted","Data":"52e73f8a4624e99a219a941c722c8333823e6a2e7925f4227fcc12c4a15e6de3"} Nov 02 18:14:14.324870 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:14.324335 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:14.345665 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:14.345620 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" podStartSLOduration=0.90729608 podStartE2EDuration="1.345606264s" totalImagesPullingTime="438.310184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:14.343390849 +0000 UTC m=+2014.442419043" watchObservedRunningTime="2026-11-02 18:14:14.345606264 +0000 UTC m=+2014.444634458" Nov 02 18:14:20.334410 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:20.334380 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fd8dffdf6-c7nwb" Nov 02 18:14:20.381272 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:20.381238 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66989884-r7kj9"] Nov 02 18:14:20.381470 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:20.381443 6343 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" podUID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" containerName="maas-api" containerID="cri-o://ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8" gracePeriod=30 Nov 02 18:14:20.589752 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:20.589673 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" podUID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.31:8443/health\": dial tcp 10.134.0.31:8443: connect: connection refused" Nov 02 18:14:21.627149 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.627125 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 18:14:21.717661 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.717583 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs\") pod \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " Nov 02 18:14:21.717661 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.717629 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9\" (UniqueName: \"kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9\") pod \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " Nov 02 18:14:21.717843 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.717684 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca\") pod \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " Nov 02 18:14:21.717843 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.717715 6343 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls\") pod \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\" (UID: \"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8\") " Nov 02 18:14:21.718061 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.718024 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca" pod "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" (UID: "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:21.719792 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.719769 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9" pod "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" (UID: "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8"). InnerVolumeSpecName "kube-api-access-h2dl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:21.719908 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.719789 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs" pod "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" (UID: "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:14:21.719908 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.719864 6343 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls" pod "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" (UID: "5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:14:21.819193 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.819162 6343 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-metrics-certs\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:21.819193 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.819186 6343 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dl9\" (UniqueName: \"kubernetes.io/projected/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-kube-api-access-h2dl9\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:21.819193 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.819196 6343 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-openshift-service-ca\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:21.819427 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:21.819206 6343 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8-maas-api-tls\") on node \"ip-10-0-141-84.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:22.367088 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.367037 6343 generic.go:356] "Generic (PLEG): container finished" podID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" containerID="ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8" exitCode=0 Nov 02 18:14:22.367254 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.367102 6343 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" Nov 02 18:14:22.367254 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.367136 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" event={"ID":"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8","Type":"ContainerDied","Data":"ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8"} Nov 02 18:14:22.367254 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.367203 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66989884-r7kj9" event={"ID":"5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8","Type":"ContainerDied","Data":"a01d4dbb00b4dbab6303eb83baa65efefef09c7dcb1f10ce5c4ff2d736e65876"} Nov 02 18:14:22.367254 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.367224 6343 scope.go:122] "RemoveContainer" containerID="ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8" Nov 02 18:14:22.375529 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.375514 6343 scope.go:122] "RemoveContainer" containerID="ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8" Nov 02 18:14:22.375764 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:14:22.375746 6343 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8\": container with ID starting with ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8 not found: ID does not exist" containerID="ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8" Nov 02 18:14:22.375808 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.375775 6343 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8"} err="failed to get container status \"ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8\": rpc error: code = NotFound desc = could not find container \"ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8\": container with ID starting with ae542c62c33b8ae1ad1686b8882ecd5cfbcdb1752942436a72a2162ff7afa2d8 not found: ID does not exist" Nov 02 18:14:22.383152 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.383132 6343 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66989884-r7kj9"] Nov 02 18:14:22.385364 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:22.385344 6343 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66989884-r7kj9"] Nov 02 18:14:24.018185 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:14:24.018144 6343 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" path="/var/lib/kubelet/pods/5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8/volumes" Nov 02 18:14:29.251764 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:14:29.251727 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:14:59.245784 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:14:59.245747 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:15:29.245065 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:15:29.244979 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:15:40.354956 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:15:40.354927 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:15:40.361001 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:15:40.360983 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:15:59.245835 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:15:59.245801 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:16:29.245167 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:16:29.245132 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:16:59.244930 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:16:59.244894 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:17:11.224223 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:17:11.224183 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zk9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:17:11.224722 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:17:11.224242 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zk9d" podUID="29129ba4-55d3-48cf-a458-20b8266a7f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:17:27.866219 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:17:27.866172 6343 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gg4rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:17:27.866993 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:17:27.866959 6343 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gg4rv" podUID="c623cbae-81a6-401d-b9e3-d40200a26ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:17:29.245564 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:17:29.245526 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:17:59.244962 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:17:59.244923 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:18:12.055554 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:12.055518 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ckhld_6c87c46e-5bc2-45d9-a00d-a880a93fa01d/manager/0.log" Nov 02 18:18:13.805942 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:13.805880 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fd8dffdf6-c7nwb_4a7dd848-560f-4f65-83eb-1a4a7873290d/maas-api/0.log" Nov 02 18:18:17.022027 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:17.021994 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vqfj2_895bdbc5-eb39-4ede-9b4b-39d32b1445b6/kuadrant-console-plugin/0.log" Nov 02 18:18:18.862754 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:18.862680 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-wmzt7_1108ab25-d875-4900-bd00-809b09003811/kube-auth-proxy/0.log" Nov 02 18:18:19.380883 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:19.380851 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-7zk9d_29129ba4-55d3-48cf-a458-20b8266a7f40/istio-proxy/0.log" Nov 02 18:18:19.891964 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:19.891938 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-gg4rv_c623cbae-81a6-401d-b9e3-d40200a26ea7/istio-proxy/0.log" Nov 02 18:18:20.223088 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:20.222988 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-nc62c_72508624-36f4-4d4a-992b-37271137a520/istio-proxy/0.log" Nov 02 18:18:29.245612 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:18:29.245566 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:18:33.426825 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:33.426788 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ckhld_6c87c46e-5bc2-45d9-a00d-a880a93fa01d/manager/0.log" Nov 02 18:18:34.848135 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:34.848102 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fd8dffdf6-c7nwb_4a7dd848-560f-4f65-83eb-1a4a7873290d/maas-api/0.log" Nov 02 18:18:37.477734 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:37.477704 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vqfj2_895bdbc5-eb39-4ede-9b4b-39d32b1445b6/kuadrant-console-plugin/0.log" Nov 02 18:18:49.192228 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:49.192197 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-wmzt7_1108ab25-d875-4900-bd00-809b09003811/kube-auth-proxy/0.log" Nov 02 18:18:49.710617 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:49.710579 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-7zk9d_29129ba4-55d3-48cf-a458-20b8266a7f40/istio-proxy/0.log" Nov 02 18:18:50.222690 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:50.222657 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-gg4rv_c623cbae-81a6-401d-b9e3-d40200a26ea7/istio-proxy/0.log" Nov 02 18:18:50.577329 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:18:50.577296 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-nc62c_72508624-36f4-4d4a-992b-37271137a520/istio-proxy/0.log" Nov 02 18:18:59.245546 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:18:59.245513 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:19:29.245624 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:19:29.245587 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:19:59.244826 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:19:59.244786 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:20:29.245012 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:20:29.244975 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:20:38.727315 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:38.727286 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xmv2b_8bd74c07-03e4-44b3-b5d3-4f06a5acb51c/global-pull-secret-syncer/0.log" Nov 02 18:20:38.939614 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:38.939587 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rkl7h_20c5900d-985b-4db3-8232-5c24ff85cc01/konnectivity-agent/0.log" Nov 02 18:20:39.012736 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:39.012658 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-84.us-east-2.compute.internal_07e33052aa2d73769dbf3107e42483d7/haproxy/0.log" Nov 02 18:20:40.385548 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:40.385521 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:20:40.392653 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:40.392632 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nvj7s_98c18ffc-40f4-4360-9095-b08f95cab47c/ovn-acl-logging/0.log" Nov 02 18:20:45.826788 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:45.826759 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vqfj2_895bdbc5-eb39-4ede-9b4b-39d32b1445b6/kuadrant-console-plugin/0.log" Nov 02 18:20:47.319226 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.319199 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/alertmanager/0.log" Nov 02 18:20:47.346247 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.346222 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/config-reloader/0.log" Nov 02 18:20:47.372821 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.372802 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/kube-rbac-proxy-web/0.log" Nov 02 18:20:47.396699 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.396679 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/kube-rbac-proxy/0.log" Nov 02 18:20:47.420980 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.420963 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/kube-rbac-proxy-metric/0.log" Nov 02 18:20:47.445090 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.445070 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/prom-label-proxy/0.log" Nov 02 18:20:47.469288 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.469270 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e765946-a617-4fc7-b66d-7e5bf4273174/init-config-reloader/0.log" Nov 02 18:20:47.552197 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.552170 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fbmsd_5701f7c4-97ba-4887-b9fe-4e3bc9941acb/kube-state-metrics/0.log" Nov 02 18:20:47.602630 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.602606 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fbmsd_5701f7c4-97ba-4887-b9fe-4e3bc9941acb/kube-rbac-proxy-main/0.log" Nov 02 18:20:47.626468 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.626442 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fbmsd_5701f7c4-97ba-4887-b9fe-4e3bc9941acb/kube-rbac-proxy-self/0.log" Nov 02 18:20:47.689521 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.689499 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-dls4w_bd8dc667-bb79-4137-845c-4cff317422ed/monitoring-plugin/0.log" Nov 02 18:20:47.722655 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.722633 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7jkqt_aec52e3c-2a94-48a7-b568-ab54be3ccb9e/node-exporter/0.log" Nov 02 18:20:47.746038 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.746021 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7jkqt_aec52e3c-2a94-48a7-b568-ab54be3ccb9e/kube-rbac-proxy/0.log" Nov 02 18:20:47.773663 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.773639 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7jkqt_aec52e3c-2a94-48a7-b568-ab54be3ccb9e/init-textfile/0.log" Nov 02 18:20:47.961791 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.961734 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q5hw5_a5a5d720-67fb-459f-823f-ad9993dec96c/kube-rbac-proxy-main/0.log" Nov 02 18:20:47.983587 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:47.983568 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q5hw5_a5a5d720-67fb-459f-823f-ad9993dec96c/kube-rbac-proxy-self/0.log" Nov 02 18:20:48.004688 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.004662 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q5hw5_a5a5d720-67fb-459f-823f-ad9993dec96c/openshift-state-metrics/0.log" Nov 02 18:20:48.044107 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.044086 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/prometheus/0.log" Nov 02 18:20:48.068364 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.068345 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/config-reloader/0.log" Nov 02 18:20:48.090666 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.090649 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/thanos-sidecar/0.log" Nov 02 18:20:48.114257 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.114241 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/kube-rbac-proxy-web/0.log" Nov 02 18:20:48.136169 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.136150 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/kube-rbac-proxy/0.log" Nov 02 18:20:48.158231 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.158216 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/kube-rbac-proxy-thanos/0.log" Nov 02 18:20:48.181183 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.181165 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_860ef26b-655c-4ce5-bfa2-17f312fa81a0/init-config-reloader/0.log" Nov 02 18:20:48.209722 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.209699 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-n2gp6_31b1e410-3fff-449a-a344-8ca439c0c9af/prometheus-operator/0.log" Nov 02 18:20:48.229536 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.229479 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-n2gp6_31b1e410-3fff-449a-a344-8ca439c0c9af/kube-rbac-proxy/0.log" Nov 02 18:20:48.299737 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.299721 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ffdb45754-7bx92_4a7e0208-a8bc-43f2-8921-5f3d0909120a/telemeter-client/0.log" Nov 02 18:20:48.323667 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.323648 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ffdb45754-7bx92_4a7e0208-a8bc-43f2-8921-5f3d0909120a/reload/0.log" Nov 02 18:20:48.345807 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.345788 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ffdb45754-7bx92_4a7e0208-a8bc-43f2-8921-5f3d0909120a/kube-rbac-proxy/0.log" Nov 02 18:20:48.579369 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.579340 6343 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2"] Nov 02 18:20:48.579718 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.579704 6343 cpu_manager.go:414] "containerMap: removing container" podUID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" containerName="maas-api" Nov 02 18:20:48.579762 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.579719 6343 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" containerName="maas-api" Nov 02 18:20:48.579814 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.579805 6343 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1d6f08-e3ad-49c8-8eb8-bf818b54a7c8" containerName="maas-api" Nov 02 18:20:48.583097 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.583076 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.591015 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.590994 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2"] Nov 02 18:20:48.678753 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.678725 6343 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4fmsl\"/\"default-dockercfg-md9st\"" Nov 02 18:20:48.680774 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.680754 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fmsl\"/\"openshift-service-ca.crt\"" Nov 02 18:20:48.680868 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.680786 6343 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fmsl\"/\"kube-root-ca.crt\"" Nov 02 18:20:48.748427 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.748406 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-proc\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.748527 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.748432 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-sys\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.748527 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.748458 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-podres\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.748604 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.748539 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85vh\" (UniqueName: \"kubernetes.io/projected/0909b88e-4891-4773-8c26-250f30b19f3d-kube-api-access-l85vh\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.748604 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.748576 6343 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-lib-modules\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849446 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849387 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-podres\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849446 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849427 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85vh\" (UniqueName: \"kubernetes.io/projected/0909b88e-4891-4773-8c26-250f30b19f3d-kube-api-access-l85vh\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849446 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849444 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-lib-modules\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849606 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849511 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-podres\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849606 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849533 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-proc\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849606 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849558 6343 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-sys\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849606 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849585 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-proc\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849606 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849585 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-lib-modules\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.849762 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.849635 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0909b88e-4891-4773-8c26-250f30b19f3d-sys\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.857710 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.857687 6343 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85vh\" (UniqueName: \"kubernetes.io/projected/0909b88e-4891-4773-8c26-250f30b19f3d-kube-api-access-l85vh\") pod \"perf-node-gather-daemonset-b6xg2\" (UID: \"0909b88e-4891-4773-8c26-250f30b19f3d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:48.896743 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:48.896726 6343 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:49.167753 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:49.167729 6343 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2"] Nov 02 18:20:49.170312 ip-10-0-141-84 kubenswrapper[6343]: W1102 18:20:49.170284 6343 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0909b88e_4891_4773_8c26_250f30b19f3d.slice/crio-f202edfd58bbd3868680ec451caf0f50ca87c8aa8bc22db09019825656d789f6 WatchSource:0}: Error finding container f202edfd58bbd3868680ec451caf0f50ca87c8aa8bc22db09019825656d789f6: Status 404 returned error can't find the container with id f202edfd58bbd3868680ec451caf0f50ca87c8aa8bc22db09019825656d789f6 Nov 02 18:20:49.185083 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:49.185037 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" event={"ID":"0909b88e-4891-4773-8c26-250f30b19f3d","Type":"ContainerStarted","Data":"f202edfd58bbd3868680ec451caf0f50ca87c8aa8bc22db09019825656d789f6"} Nov 02 18:20:50.190454 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:50.190422 6343 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" event={"ID":"0909b88e-4891-4773-8c26-250f30b19f3d","Type":"ContainerStarted","Data":"f38803d64b91a4813ba39cae7cfd264ecb8c1e7d550dfd437133d8b8aca1523c"} Nov 02 18:20:50.190844 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:50.190475 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:50.209109 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:50.209040 6343 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" podStartSLOduration=2.209024342 podStartE2EDuration="2.209024342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:50.208852936 +0000 UTC m=+2410.307881208" watchObservedRunningTime="2026-11-02 18:20:50.209024342 +0000 UTC m=+2410.308052538" Nov 02 18:20:52.738231 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:52.738209 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgw6g_6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/dns/0.log" Nov 02 18:20:52.760754 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:52.760728 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgw6g_6440b09a-ee4a-4ffa-8102-da0d1d3e10bb/kube-rbac-proxy/0.log" Nov 02 18:20:52.945871 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:52.945850 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zb589_ee56fd64-8479-45f4-b098-16208b7ced9f/dns-node-resolver/0.log" Nov 02 18:20:53.562938 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:53.562911 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-25nk6_12700290-2871-451f-9f86-3ea2830de15d/node-ca/0.log" Nov 02 18:20:56.205064 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:56.205021 6343 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-b6xg2" Nov 02 18:20:57.816504 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:57.816469 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-wmzt7_1108ab25-d875-4900-bd00-809b09003811/kube-auth-proxy/0.log" Nov 02 18:20:57.903552 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:57.903526 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-7zk9d_29129ba4-55d3-48cf-a458-20b8266a7f40/istio-proxy/0.log" Nov 02 18:20:58.004530 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:58.004505 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-gg4rv_c623cbae-81a6-401d-b9e3-d40200a26ea7/istio-proxy/0.log" Nov 02 18:20:58.064452 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:58.064423 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-nc62c_72508624-36f4-4d4a-992b-37271137a520/istio-proxy/0.log" Nov 02 18:20:59.244935 ip-10-0-141-84 kubenswrapper[6343]: E1102 18:20:59.244904 6343 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kk5x9" containerName="extractor" Nov 02 18:20:59.932477 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:20:59.932452 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88nck_40498a67-6c10-479b-b130-3c97cdd76774/serve-healthcheck-canary/0.log" Nov 02 18:21:00.712883 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:00.712856 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kk5x9_72cc37d8-2509-4902-8494-415871c9db22/kube-rbac-proxy/0.log" Nov 02 18:21:00.736886 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:00.736868 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kk5x9_72cc37d8-2509-4902-8494-415871c9db22/exporter/0.log" Nov 02 18:21:00.769968 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:00.769947 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kk5x9_72cc37d8-2509-4902-8494-415871c9db22/extractor/0.log" Nov 02 18:21:02.852407 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:02.852374 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-wmzt7_1108ab25-d875-4900-bd00-809b09003811/kube-auth-proxy/0.log" Nov 02 18:21:02.910857 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:02.910829 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-7zk9d_29129ba4-55d3-48cf-a458-20b8266a7f40/istio-proxy/0.log" Nov 02 18:21:02.963774 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:02.963749 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-gg4rv_c623cbae-81a6-401d-b9e3-d40200a26ea7/istio-proxy/0.log" Nov 02 18:21:02.998352 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:02.998301 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-nc62c_72508624-36f4-4d4a-992b-37271137a520/istio-proxy/0.log" Nov 02 18:21:03.651376 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:03.651344 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88_52bf20cf-0da5-496a-ab03-ff53cfc7587f/manager/0.log" Nov 02 18:21:03.687313 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:03.687287 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6f4dd5dc4d-8wr88_52bf20cf-0da5-496a-ab03-ff53cfc7587f/copy-manifests/0.log" Nov 02 18:21:05.040833 ip-10-0-141-84 kubenswrapper[6343]: I1102 18:21:05.040808 6343 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-7b476c9985-ttxw6_773f1045-d567-4a52-b84d-cd01dcf02b94/manager/0.log"